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Brief Summary of Intervention and Evaluation Design 


Description of Intervention 

With the support of an 13 development grant, Oakland Unified School District (OUSD), together 
with the College Board, implemented Oakland Accelerates, a program intended to bring the College 
Board’s EXCELerator district-wide process for college readiness to OUSD. The desired outcome 
was to increase district-wide capacity to support each and every student to graduate from high 
school with the knowledge and skills to be successful in college and postsecondary courses of study. 


Oakland Accelerates began with a baseline District Diagnostic process in 2010-11, designed to 
analyze gaps in a district’s college readiness infrastructure by reviewing five critical components of 
college readiness: 1) District Infrastructure; 2) Curriculum Coherence; 3) Assessments That Inform; 
4) Student Academic Support; and 5) Student Family Support. The results of the district-wide 
diagnostic guided the development of a road map for change in OUSD overall and within specific 
secondary schools. An Oakland Accelerates Steering Committee, supported by Project Management 
and Data Teams, identified, agreed on, and prioritized the set of College Readiness initiatives and 
actions (henceforth called the Management Plan) to achieve the goals of Oakland Accelerates. 


This Steering Committee engaged in ongoing cycles of inquiry to inform progress and modifications 
for the duration of the project. Key intervention activities included: 1) professional development for 
AP teachers, leadership, and counselors to support instruction and development of a college-going 
culture; 2) support and coaching for College Readiness Specialists to work throughout OUSD 
providing resources and other expertise to effectively change the practices of adults at every level of 
the district, equipping them to provide a rigorous education to every OUSD student, increase 
student interest in and access to college, and better support students’ and parents’ needs; 3) technical 
assistance to develop policy and infrastructure to support college going culture; 4) provision of key 
resources to support teachers, students, and families to build an understanding of college 
preparation; and 5) monitoring and managing the intervention to immediately address challenges 
and allow for midway course corrections. ' 


Description of Impact Studies Conducted 

In accordance with Department of Education 13 grant requirements, the Oakland Accelerates 
evaluation plan’ was formally reviewed and approved in a peer-review process. (See Table 1 for an 
overview of the evaluation design.) In addition, HTA received one-on-one assistance from a TA 
provider from Abt Associates’. 


Two quasi-experimental impact studies were conducted to evaluate the impact of Oakland 
Accelerates on student academic performance outcomes at the school-level. Impact Study 1 utilizes 
a nonequivalent comparison group design to examine the effect of the Oak/and intervention process 
on high school student academic outcomes relating to college placement and college entrance exam 
participation and performance (AP and SAT exam participation and performance) compared to 


! The Final Implementation Report of Oakland Accelerates: January 2012 - December 2015 provides a more 
comprehensive and detailed description of how Oakland Accelerates was implemented. A copy can be requested 


from HTA. 
? Acopy of the approved Oakland Accelerates 2013 evaluation plan can be requested from HTA. 
3 See http://ies.ed.gov/ncee/projects/evaluation/assistance_ita.asp for more details on the i3 national evaluation. 
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schools not receiving the treatment. Specifically, the evaluators compared school-level outcomes 
from students in treatment and comparison schools that share similar demographic and academic 
outcome characteristics. Impact Study 2 examines the impact of the Oakland Accelerates 
intervention on high school student college and career preparation. A one-group pretest-posttest 
design compared outcomes at the school level for differences in graduation rates, college enrollment 
without need for remediation, a-g course completion, and college enrollment. 


While there was not any specific eligibility criteria for students to be included in the intervention 
(.e., all students within the targeted OUSD schools were indirectly served by the intervention) there 
were some exclusion criteria that required elimination of particular schools from the study. 
Specifically, we excluded continuation/alternative/charter high schools, which comprised a unique 
student population that does not adequately generalize to other local student populations. 


The evaluation was conducted so as to meet standards for independence. Specifically, findings 
reported to the National Evaluation 13 (NEi3) team were not subject to the approval of the project 
developer/grantee (ie., College Board & OUSD). 


Methodology of Impact Study 1 


The first study utilizes a nonequivalent comparative group design to examine the effects of the 
Oakland Accelerates process on high school students in treatment schools compared to similar 
students in untreated control schools. 


Research Question 

RQ1 generates preliminary evidence of the promise of the intervention for improving student 
outcomes: participation and performance on college entrance and placement tests. All analyses were 
conducted at the school-level, therefore the intervention sample comprised all schools and students 
from the traditional OUSD high schools and the comparison sample comprised all schools and 
students from the traditional comparison high school district. Findings from the study can be 
generalized to students attending a mid-sized urban district with high poverty rates and low rates of 
student achievement and college attendance. 


RQI!1: Following the completion of the Oakland Accelerates intervention, are OUSD students’ 
participation rates and performance on college entrance and placement tests higher than those of 
students at untreated comparison schools? 


Data collection for evaluation of impacts 
1. Outcomes: College entrance and placement exam participation rates 
e AP examinations participation (Y%): The Advanced Placement Program® (AP) has been a 

unique collaborative effort among motivated students, dedicated teachers, and committed 
high schools, colleges, and universities. AP includes curriculum guidelines, professional 
development opportunities for teachers, and the examinations. At the time the study began, 
there were 33 courses and examinations in 22 subject areas. Each AP exam was 
administered once during a two week period in May with the vast majority of test takers 
being administered the same form. Following that, each exam was offered once during a 
three-day late testing period with a different form. The scoring of the exam was completed 
during the first three weeks of June. The current study examines whether there was a 
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differential change from baseline to follow-up in AP participation between students in 
OUSD schools compared to students in the control group. Data for this outcome was 
requested directly from the OUSD and SUSD Research departments. We operationalized 
the AP participation rate as, the number of students in all grades for whom College Board 
reported AP exam scores divided by Grade 11 and 12 CBEDS enrollments at each school, 
weighting each school’s percentage to account for substantially different school sizes within 
OUSD and SUSD. As such, a student may participate in 4 exams, but only be counted once. 
Number of exam takers was not be used as an outcome variable as it is likely dependent on 
school size. Outcome comparisons were made between baseline (2011-12) and final 
program year (2014-15) for both groups. 

SAT participation (%): The Scholastic Aptitude Test (SAT) is a standardized test that 
assesses the critical reading, mathematics, and writing skills that students need to be 
successful in college. Each of the three sections that comprise the SAT Reasoning Test had a 
possible scale score of 800 points. SAT test results represent one factor considered by many 
colleges and universities in making admissions decisions. The SAT is owned, published, and 
developed by the College Board. At the time this study began, the SAT was offered seven 
times a year in October, November, December, January, March (or April, alternating), May, 
and June. Data for this outcome was requested directly from the OUSD and SUSD Research 
departments. We operationalized this variable as the number of students who participated in 
the SAT divided by Grade 12 CBEDS enrollment at each school. Outcome comparisons 
were made between baseline (2011-12) and final program year (2014-15) for both groups. 


2. Outcomes: College entrance and placement exam performance 


AP examinations performance (%): AP exam grades are reported on a 1—5 scale. Students’ 


raw scores on the multiple-choice (MC) and constructed response (CRS) items are weighted 
and combined. The weighted raw composite score is then converted to a grade on the AP 5 
point scale. The AP grade qualification definitions are: 5=Extremely well qualified; 4 =Well 
qualified; 3 =Qualified; 2 =Possibly qualified; and 1 =No recommendation. Internal 
consistency reliability estimates for total raw scores range from .85 to .94 across 
examinations and estimates of classification accuracy as reported are generally around .90. 
The reliability estimates for the same examination tend to be very close from year to year. 
So, at the composite level the AP exams seem to exhibit reasonable reliability estimates. Data 
for this outcome were requested directly from the OUSD and SUSD Research departments. 
Because students may have taken multiple AP exams, we operationalized this variable as the 
number of students who received an AP score of 3 or above on at least one exam divided by 
divided by Grade 10, 11, and 12 CBEDS enrollments at each school, weighting each school’s 
percentage to account for substantially different school sizes within OUSD and SUSD. 
Outcome comparisons were made between baseline (2011-12) and final program year (2014- 
15) for both groups. The impact estimation also controlled for school size. 

SAT performance (mean): SATs test in three content areas: critical reading, mathematics and 
writing. Scores are calculated for each content area, as well as a sum score totaling each 
section. Data for this outcome was requested directly from the OUSD and SUSD Research 
departments. Because students may have taken multiple SAT exams in a year, we selected 
the highest total score for each student. Then we calculated this variable as the mean SAT 
score at each school. Outcome comparisons were made between baseline (2011-12) and 
final program year (2014-15) for both groups. 
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Over-alignment occurs when the treatment group is exposed to the outcome measure but the 
compatison group is not. The outcomes described above are not over-aligned with the intervention. 
In the current study, no intermediary assessments were administered to the treatment group, which 
would overlap with the outcome measures and bias the outcome. 


3. Independent Variables 

School level covariates were included in order to control for significant between-school variations. 
This included a baseline measure of academic achievement, and two demographic characteristics, 
school size (total number of students), and English Language Learners (/ ELL students). 

e School-level Academic Performance Index (API) score (mean): Calculated by the California 
Department of Education (CDE), the API is a single number of school academic 
performance and improvement of schools based on student performance on the state-wide 
tests in reading and math. It is used for accountability purposes in accordance with 
California’s Public Schools Accountability Act (PSAA). API scores range from 200 to 1000, 
with 800 as the state’s target for all schools to meet. Base and Growth API scores ate 
calculated annually and posted online by the CDE for each school; generally in the fall 
semester following the academic school year. (Change in the API is the difference between 
the Base API and Growth API within a reporting cycle.) API scores were retrieved from 
CDE Dataquest. 

e = School-level Demographic Characteristics (# and %): School size refers to the total number 
of freshmen, sophomores, juniors and seniors enrolled in the school. Percentage of students 
that are ELL refers to the total number of Grade 9-12 students designated as English 
Language Learners divided by the total number of Grade 9-12 CBEDS enrollments. English 
learner students are those students for whom there is a report of a primary language other 
than English on the state-approved Home Language Survey and who, on the basis of the 
state approved oral language assessment procedures (grades kindergarten through grade 
twelve) and literacy (grades three through twelve only), have been determined to lack the 
clearly defined English language skills of listening comprehension, speaking, reading, and 
writing necessary to succeed in the school's regular instructional programs. 


While the inclusion of additional covariates would control for between school differences, the 
current sample size (n=12) did not allow for this. Covariate data were retrieved from CDE’s 
Dataquest website for the baseline year (2011-12). All covariates were measured and recorded 
before treatment began. 


Statistical analysis of impacts on students 
Outcomes for this particular impact study were assessed for 11° and 12" grade students who 
attended the high schools served by the intervention (or comparison) during the baseline yeat* 


4In 2011-12, thirteen OUSD schools were included in this study: 1) Business and Information Technology High, 2) 
Castlemont High, 3) Coliseum College Prep Academy, 4) College Preparatory and Architecture Academy, 5) East 
Oakland School of the Arts, 6) Leadership Preparatory High, 7) LIFE Academy, 8) Mandela High, 9) McClymonds 
High, 10) Media College Preparatory, 11) Oakland High, 12) Oakland Technical High, and 13) Skyline High. The 
Castlemont campus comprised four small high schools: 1) Business and Information Technology High, 2) East Oakland 
School of the Arts, 3) Leadership Preparatory High, and 4) Castlemont High School. The Fremont campus comprised 
three independent schools: 1) College Preparatory and Architecture Academy, 2) Mandela High, and 3) Media College 
Preparatory. 
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(2011-12) and final program year °(2014-15) of the study, regardless of gender, ethnicity, or SES. 
The analysis was conducted at the school level to determine whether mean differences in outcomes 
exist between treatment and control schools at study end, controlling for pretest outcomes and 
posttest school-level variables expected to impact the posttest outcome. All student level data was 
ageregated to form a school mean for each school for each outcome and covariate. 


The design has several confounds including threats to internal validity associated with a quasi- 
experimental design (e.g. selection, Hawthorne effect) and the N=1 problem (only one district is 
participating in the intervention). Given the lack of statistical power, it was difficult to determine an 
effect with precision. Because the unit of assignment (district) is relatively large, it was not 
logistically or financially possible to recruit additional treatment or control districts. We believed 
this design, as a development study, would indicate whether the intervention, showed evidence of 
promise. 

The study compates differences between the treatment and comparison school’s student outcomes 
over the course of the study. Specifically, this includes data from students who attend the following 
academic school years: Baseline Intervention Year (2011-12 academic school year) and Intervention 
Year 3 (2014-15 academic school year). It is important to understand the decision behind selecting 
the 2011-12 year as the baseline year. While funding for the intervention began in early 2012, the 
activities at this point consisted primarily of meetings between the key stakeholders (College Board 
and OUSD leadership) which lay the groundwork for future communication. Much of the work 
done in 2011-12 was a precursor to the delivery of the intervention; its primary focus was planning 
the activities needed for the intervention. The process started with the District Diagnostic in 
November 2011, and continued with district leadership planning activities to review the Diagnostics 
results and develop a management plan. Second, teacher trainings, capacity building, and policy 
guidance meetings occurred in the summer of 2012 and consequently do not have an impact on 
students enrolled in the 2011-12 school year. 


By the end of the program (2015-16), all students who attended OUSD had been exposed to 
teachers, college readiness specialists, counselors, and other faculty and staff who received the 
treatment. Individual students, however, had varying degrees of exposure. For example in the final 
year of the study, some seniors had the benefit of being in the district for the entirety of the 
intervention whereas others had only been in the district for the past year three years as they began 
high school when the study was already in its first year. Similarly, student level differences exist as 
students transferred in and out of the district. Nevertheless, this issue is secondary as student 
mobility is exogenous to the intervention. First, we believe that the sample of post-test students is 
representative of the pre-test sample as many students in Oakland move to other schools within the 
district. And second, the goal of the intervention is to improve college readiness at the district level, 
regardless of mobility. Thus, college readiness culture/outcomes would be better following the 
intervention. 


To address RQ1, four confirmatory contrasts° were identified for the following two domains: 


>In 2012-13, the Fremont campus re-opened as a single high school, Fremont High, absorbing the three small schools 
that were previously on the site. In the same year, the Castlemont campus also re-opened as a single high school, 
combining the four independent high schools that were previously on the site into Castlemont High. Thus, the twelve 
high schools in the study were reduced to eight from the baseline year to the first year of programming. However, it is 
important to note that the number of students remained essentially the same (aside from normal attrition within OUSD 
high schools) at both campuses. 

6 See Appendix for Registration of Contrasts 
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Domain 1: Participation in College entrance placement and performance exams 
1. AP participation (%) 
2. SAT participation (0) 
Domain 2: Performance on college entrance and placement exams 
1. AP score (%) 
2. SAT score (mean) 


Confirmatory contrasts were made between OUSD and SUSD schools in the final program year of 
the grant for each of the outcomes noted above. Each contrast controlled for baseline measures of 
the outcome (e.g. performance at baseline). Additionally, we controlled for other independent 
variables that may have co-varied with the outcome at the school level. Covariates included: school 
size (#), API score (mean), and ELL (%). All covariates except the treatment variable were centered 
at the mean. Four linear regressions were conducted to predict the effect of the 
treatment/intervention on AP and SAT participation rates, and AP and SAT performance scores. 
Schools were treated as the unit of analysis. Specifically, baseline outcome scores, covariates, and 
treatment group were regressed on the identified outcome. 


Thus, we estimated the following linear regression model to estimate intervention impacts: 


AP/SAT = Bo + Bi(X1) + Ba(X2) + B3(X3) + BaCX4) + (Ts); 


whereby X1= baseline outcome measure (e.g., mean SAT score, etc.); 
X2= API score; 

X3=# of students in the school; 

X4= % ELL, and 

Tx= treatment group (1=treatment group; 0= control group). 


Covariates selected for inclusion in the analysis were theory driven and chosen a priort. Specifically, 
we included items that were known to correlate with academic achievement and items that indicate 
school demographics. 


Additional exploratory subgroup analyses were conducted for this research question by 
race/ethnicity and gender. For race, in particular, subgroups of African-American, Hispanic, and 
ELL students were separately analyzed as the subsample sizes permitted. In cases where the 
subgroup of interest was 10 students or fewer in a particular school, these schools were dropped 
from the analyses. 


Findings of Impact Study 1 


Baseline Equivalence 

The intervention population for this study comprised almost 7,000 G11-12 students at eight 
traditional high schools in OUSD’ and 11,000 G11-12 students at eleven traditional high schools at 
a comparable school district in 2011-12 and 2014-15. Table 1 describes the student population at 


7In 2010-11, there were twelve comprehensive OUSD high schools. In the baseline year (2011-12), these twelve high 
schools were re-opened as thirteen high schools. (The Castlemont campus added a small high school recruiting students 
from the other three small high schools on the site.) In 2012-13, these thirteen high schools were re-opened as eight 
high schools, absorbing a number of small high schools and independent schools into larger high schools. The total 
number of students remained essentially the same. The schools continued as eight high schools through 2014-15. 
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the treatment and comparison school districts in 2011-12 (pretest) and 2014-15 (posttest). Table 2A 
and 2B describe the pretest outcomes for the treatment and comparison school districts in 2011-12. 


OUSD is a unique school district in many ways — its size in terms of number of schools and 
students, its urban context, and its particular student demographics. It was very difficult to find a 
school district with many of the same characteristics. In the end, we chose a California school 
district® with qualitatively as many characteristics in common with OUSD as possible — however, the 
school district still differs in several key respects. The factors that were significantly different at 


baseline were: 


e Percentage and mean number of Special Education students per school (OUSD has a higher 
percentage and mean number); 


e Percentage of African-American students per school (OUSD has a higher percentage); 


e Percentage and mean number of White students and Other students (Comparison has higher 
percentage and mean number of both groups); 


e Weighted percentage of students taking SAT and AP tests (OUSD has higher percentage for both 


tests); and 


e Weighted percentage of students scoring a 3, 4, or 5 on any one AP test (Comparison has higher 


percentage). 


Table 1. Study School Profiles of G11-G12 Students for OUSD & Comparison, 2011-12 


Oakland USD Comparison 
(N=12 schools) (N=11 schools) 
# schools meeting AYP goals? 0 1 
% per high school % SD % SD t-statistic p-value 
English language learners 15.5 8.0 13.6 8.5 532 .600 
Special education students 17.9 7.3 0.6 0.8 8.181 .000 
Ethnicity 
African American 36.8 21.6 16.6 6.0 3.112 .008 
Latino 42.6 24.1 32.0 9.3 1.418 177 
Asian/Pac Islander 15.9 14.4 21.5 14.0 -.948 354 
White 3.1 5.9 26.3 15.8 -4.574 .001 
Other 1.7 1.2 3.7 1.5 -3.605 .002 
Mean # per high school Mean SD Mean SD t-statistic p-value 
G11-12 Students 279.1 303.8 548.5 441.3 -1.691 .109 
English language learners 35.7 35.2 94.4 109.0 -1.706 114 
Special education students 49.8 55.3 6.0 7.5 2.712 .020 
Ethnicity 
African American 95.9 101.6 86.1 70.3 .267 .792 
Latino 81.8 57.5 175.2 145.8 -1.988 .069 
Asian/Pac Islander 76.2 128.5 153.7 165.9 -1.260 .228 
White 20.8 45.8 114.2 107.2 -2.673 .019 
Other 4.4 6.2 19.4 17.7 -2.650 021 


8 The comparison school district that we selected graciously provided us with student-level data in exchange for not 


identifying the school district in any publicly available documents. 


° Determination of AYP (adequate yearly progress) was a requirement of the federal Elementary and Secondary 
Education Act (ESEA). Possible values are “yes” or “no”. A “yes” is recorded only if the school met all of its AYP 
criteria, including requirements for numerically significant student groups. 
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Table 2. Weighted Differences?® in SAT/AP Outcomes for OUSD & Comparison, 2011-12 


Oakland USD Comparison 
(N=12) (N=11 schools) 

Test Participation N %/p SD N %/p SD t-statistic p-value 
% taking SAT (of G12)" 12 57.2 iD 11 42.4 13.0 2.813 .010 
% taking 1+ AP test (of G11-12) 12 31.7 9.9 11 15.0 13.2 3.466 .002 
Test Performance?” 

SAT Combined Score 11 1364.2 154.8 10 1427.0 163.6 -.902 .378 
% w/ AP score of 3-5 11 38.8 17.3 8 65.4 18.4 -3.213 .005 


Participation in college entrance placement and performance exams 

Tables 3 and 4 present the results of the two confirmatory contrasts for the first domain, 
participation in college entrance placement and performance exams. These results indicate that 
OUSD schools had a significantly greater percentage of students who participated in the SAT in 
2014-15 in contrast to the comparison district school (p=.033), after controlling for SAT 
participation in 2011-12, AYP scores, school size and proportion of ELL students. 


Table 3. Summary of Unweighted OLS Regression Analysis for Variables Predicting Posttest SAT Participation 
(N=19) 


Unstandardized Standardized 95% Conf. Interv. 
Coeff. Coeff. 

Vanable B Std. Error B Lower Upper p-value 
Treatment group (1=OUSD) .262 .110 .623 .025 .498 .033 
Pretest SAT Participation (%) .602 .285 .382 -.013 1.216 .054 
Posttest AYP Scores (1=Met goals) -.012 .089 -.028 -.204 .181 .897 
Posttest School size (#4, G11-12) .000 .000 -.123 .000 .000 484 
Posttest ELL Students (%, G12) -.207 344 -.119 -.951 537 559 
(Intercept) .070 .162 -.281 .420 .675 
R? .737 

Adjusted R? .636 


Note: All tests were two-tailed. 


However, there was no comparable finding for AP test participation in 2014-15. That is, there was 
no statistically significant difference between OUSD schools and the comparison district schools in 


‘0 Means/percent were not calculated for schools with <10 students in the sub-population. Weights were calculated by 
school size for participation, and by test-taker size for performance. 

" Student-level SAT scores were not available for the comparison for 2011-12, so we were unable to match students 
with test-takers and verify if they were students of the school where they took the test. Also we could not confirm 
whether the number of students taking the SAT was unduplicated counts. (The College Board provided us with counts 
and mean combined scores by school.) Therefore, SAT participation rates & means here may be an over-representation 
of true participation and scores. 

'2 Test performance means and rates were only calculated among test-takers. 
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the AP test participation rate in 2014-15, after controlling for AP test participation in 2011-12, AYP 
scores, school size and proportion of ELL students. 


Table 4. Summary of Unweighted OLS Regression Analysis for Variables Predicting Posttest AP Test Participation 
(N=19) 


Unstandardized Standardized 95% Conf. Interv. 
Coeff. Coeff. 

vonanle B Std. Error B Lower Upper p-value 
Treatment group (1=OUSD) -.011 -110 -.033 -.249 .226 .918 
Pretest AP Test Participation (%) .836 .276 .728 .239 1.433 .010 
Posttest AYP Scores (1=Met goals) -.048 .098 -.140 -.260 .165 .636 
Posttest School size (#4, G11-12) .000 .000 -.483 -.001 .000 .091 
Posttest ELL Students (%, G12) .275 .358 .193 -.498 1.047 457 
(Intercept) .212 123 -.054 .478 .109 
R? 544 

Adjusted R? 369 


Note: All tests were two-tailed. 


Taken together, these findings suggest that Oakland Accelerates may have had an impact on 
increasing the percentage of OUSD students taking the SAT exam, but did not have a comparable 
impact on increasing the percentage of OUSD students taking the AP exams. 


Performance on college entrance placement and performance exams 

Tables 5 and 6 present the results of the two confirmatory contrasts for the second domain, 
performance on college entrance and placement exams. These results indicate that after controlling 
for SAT combined mean score in 2011-12, AYP scores, school size and proportion of ELL 
students, the SAT combined mean score in 2014-15 was not significantly different than that of the 
compatison district schools. It is worth noting that increasing the percentage of OUSD students 
taking the SAT may have perversely lowered SAT mean combined scores. If less academically 
prepared students are encouraged to take the SAT, then it would not be surprising that the scores 
would not increase, but decrease. In fact, the negative beta value for the treatment group in Table 5 
suggests that the SAT mean combined score actually declined from 2011-12 to 2014-15 for OUSD 
schools compared to the comparison district schools. 
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Table 5. Summary of Unweighted OLS Regression Analysis for Variables Predicting Posttest SAT Performance 


(N=18) 
Unstandardized Standardized 95% Conf. Interv. 
Coeff. Coeff. 
Vana B Std. Error B Lower Upper p-value 
Treatment group (1=OUSD) -12.654 50.223 -.037 -122.081 96.772 .805 
Pretest SAT Combined Score (Mean) 775 .200 .739 338 1.211 .002 
Posttest AYP Scores (1=Met goals) 89.300 57.080 .262 -35.067 213.667 .144 
Posttest School size (#, G12) .081 .078 .166 -.088 .250 .317 
Posttest ELL Students (%, G11-12) 14.250 237.499 010 -503.216 531.717 .953 
(Intercept) 209.552 256.131 -348.508 767.613 429 


R2 


Adjusted R? 


.893 
.849 


Note: All tests were two-tailed. 


Moreover, the percentage of AP test-takers earning a 3 or higher on the AP test in 2014-15 was not 
significantly different between OUSD schools and comparison district schools, after controlling for 
percentage of AP test-takers earning 3 or higher on the AP test in 2011-12, AYP scores, school size, 


and proportion of ELL students. 


Table 6. Summary of Unweighted OLS Regression Analysis for Variables Predicting Posttest AP Test Score >2 


(N=15) 
Unstandardized Standardized 95% Conf. Interv. 
Coeff. Coeff. 

vanaule B Std. Error B Lower Upper p-value 
Treatment group (1=OUSD) -.187 .163 -.345 -.555 .180 .279 
Pretest AP Test Score >2 (%) 572 215 579 .085 1.058 .026 
Posttest AYP Scores (1=Met goals) .090 .159 .167 -.271 .450 588 
Posttest School size (#4, G11-12) .000 .000 311 .000 .001 .235 
Posttest ELL Students (%, G12) .697 587 .310 -.630 2.024 .265 
(Intercept) .038 .227 -.474 550 .870 
R? .656 

Adjusted R? 465 


Note: All tests were two-tailed. 


In sum, these findings suggest that Oakland Accelerates did not have the expected impact on OUSD 


students in terms of increasing performance on either the SAT or AP exams. In fact, it is possible 
that increasing the percentage of OUSD students taking the SAT in 2014-15, resulted in a decrease 


or no change in the mean SAT combined score for OUSD students in 2014-15. 
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Exploratory Impact Analyses for Sub-groups 

Baseline Equivalence 

Unfortunately, SAT data broken down by sub-groups was not available for the comparison district 

schools in 2011-12; the individual-level data was sent directly to the schools and not saved, and the 

College Board does not maintain individual-level records for more than 3 years. Therefore only AP 
participation and performance outcomes were compared in the exploratory analyses. 


Table 7. Weighted Differences*? in AP Outcomes by Sub-groups for OUSD & Comparison, 2011-12 


Oakland Comparison 
(N=12) (N=11 schools) 

% taking 1+ AP test (G11-12) N % SD N % sD t-statistic p-value 
Among African-American 11 16.4 6.1 11 4.6 6.4 4.434 .000 
Among Hispanic/Latino/a 11 27.5 1.1 11 10.6 10.2 3.736 001 
Among English language learners 10 11.7 75 8 3.1 1.4 3.171 .006 

% w/ AP score of 3-5 on any 1 test 
Among African-American 5 20.8 13.6 2 56.0 19.5 -2.811 .038 
Among Hispanic/Latino/a 11 45.2 18.4 6 74.6 10.4 -3.498 .003 
Among ELL” 2 48.2 2.3 0 -- -- - - 


Table 7 presents the baseline equivalence on the AP outcome measures among the three subgroups 
of interest. It is evident that the two districts are significantly different for participation and 
performance among all subgroups of interest. In general, OUSD schools had statistically higher 
percentages of African-American, Hispanic, and ELL students taking one or more AP tests than the 
compatison district schools. Conversely, the comparison district schools had statistically greater 
percentages of African-American and Hispanic AP test-takers who scored 3 or higher on any AP 
test. 


It is worth noting that both districts had substantial missing data on AP performance — as many 
schools did not have enough students within the subgroup taking the AP test (S10) to be able to 
calculate the percentage of test-takers with scores of 3 or higher. We were unable to compare OUSD 
and the comparison district among ELL students at all. Therefore, we decided to only analyze the 
AP participation sub-group data. 


Participation in college performance exams 

Table 8-10 show the separate results of the exploratory subgroup analyses for African-American, 
Hispanic or ELL students. These analyses indicate that AP participation rates for African-American, 
Hispanic or ELL students in 2014-15 was not significantly different at the .05 level for OUSD 
schools compared to the comparison district schools, after controlling for their respective subgroup 
AP patticipation rate in 2011-12. However, the AP participation rate for Hispanic students in 2014- 
15 was significantly higher than the comparison at the 0.10 level (p=.09). Therefore, while there is 
some evidence that Oakland Accelerates may have had an impact on Hispanic students’ increasing 
participation in AP tests from 2011-12 to 2014-15; for the most part, there is no evidence that the 
program had an impact on increasing participation for African-American or ELL students. 


'3 Means/percent were not calculated for schools with <10 students in the sub-population. Weights were calculated by 
school size for participation, and by test-taker size for performance. 
4 ¢ cannot be computed because at least 1 of the groups is empty 
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Table 8. Summary of Unweighted OLS Regression Analysis for Variables Predicting African-American Posttest AP 


Test Participation (N=18) 


Unstandardized Standardized 95% Conf. Interv. 
Coeff. Coeff. 

Vana B Std. Error B Lower Upper p-value 
Treatment group (1=OUSD) .073 .053 .327 -.041 .186 .192 
Pretest A-A AP Test Participation (%) .386 .290 319 -.231 1.004 .202 
(Intercept) .059 .034 -.014 131 105 
R? .297 

Adjusted R2 .204 


Note: All tests were two-tailed. 


Table 9. Summary of Unweighted OLS Regression Analysis for Variables Predicting Hispanic Posttest AP Test 


Participation (N=18) 


Unstandardized Standardized 95% Conf. Interv. 
Coeff. Coeff. 
B td. E L -val 
Verse Std. Error B ower Upper p-value 
Treatment group (1=OUSD) .081 .092 .220 -.115 .276 .090 
Pretest Latino AP Test Participation 
(%) .620 .342 453 -.109 1.348 118 
(Intercept) .099 .059 -.028 225 393 
R? .368 
Adjusted R2 284 


Note: All tests were two-tailed. 


Table 10. Summary of Unweighted OLS Regression Analysis for Variables Predicting ELL Posttest AP Test 


Participation (N=12) 


Unstandardized Standardized 95% Conf. Interv. 
Coeff. Coeff. 

Varahic B Std. Error B Lower Upper p-value 
Treatment group (1=OUSD) .025 .098 .081 -.197 .247 .804 
Pretest ELL AP Test Participation (%) 1.504 .707 .670 -.096 3.103 .062 
(Intercept) .034 .050 -.079 146 515 
R? .530 

Adjusted R? 426 


Note: All tests were two-tailed. 
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Methodology of Impact Study 2 


The second study used a one-group pretest-posttest design to assess the impact of the Oakland 
Accelerates process on OUSD high school student college preparation, including high school 
graduation, college readiness, college enrollment without need for remediation, and college 
enrollment. 


Research Questions 

RQs 2 and 3 were proposed to generate preliminary evidence of the promise of the intervention for 
improving student outcomes on student college preparation for all students and for traditionally 
under-represented students. All analyses were conducted at the school-level, therefore the 
intervention sample comprised all schools and students from the eight traditional OUSD high 
schools which was the population identified for the i3 grant. Although this study uses a pretest- 
posttest design, we believe that the results of the study can be generalized to other large districts that 
serve low-income, under-represented, African American and Latino youth in inner cities. 


RQ2: What is the impact of the Oakland Accelerates process on student college preparation? 
Specifically are OUSD students at program end better prepared for college than OUSD students at 
baseline? 

RQ3: What is the impact of the Oakland Accelerates process on student college preparation prior to 
and following the intervention on traditionally under-represented students, including African- 
American males, Latinos, and English Learners? Specifically ace OUSD under-represented students 
at program end better prepared for college than OUSD under-represented students at baseline? 


Data collection for evaluation of impacts 
1. Outcomes: High school graduation rates (%) 

e Graduation (Y%): In California, graduation rates are calculated by the California Department 
of Education only; school districts are not allowed to calculate the graduation rates 
themselves. CDE calculates graduation rates for each school using the following formula: 
Number of students in the cohort who earned a regular high school diploma by the end of 
the fourth year divided by number of first-time Grade 9 students in year one (starting 
cohort) plus students who transfer in, minus students who transfer out, emigrate, or die 
during school years one, two, three, and four. School-level data for this outcome was 
retrieved directly from the CDE Dataquest website. Outcome comparisons were made 
between the graduation rate calculated for the class of 2012 as the baseline (2011-12) and the 
graduation rate calculated for students who should graduate in the final program year (2014- 
15) for all eight schools within OUSD. 


2. Outcomes: College readiness (%) 

e A-Gcourse completion (%): The University of California (UC) and California State 
University (CSU) systems require entering freshmen to complete certain courses in high 
school. These courses are called the "a-g" requirements because of the letter each subject 
atea is assigned: "a" for History/Social Science, "b" for English, etc. Data for this 
outcome was requested directly from the OUSD Research department. We 
operationalized this outcome as the percent of Grade 12 students completing all a-g 
requirements by taking the number of students in each school who meet the 
requirements and dividing by the Grade 12 CBEDS enrollments. Outcome comparisons 
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were made between the percentage of grade 12 students who met the a-g course 
requirements in the baseline year (2011-12) and the percentage of grade 12 students who 
met the a-g course requirements in the final program year (2014-15) for all eight schools 
within OUSD. 


3. Outcomes: Waived remediation for college English (%); and college Math (%) 

Two composite outcome variables for waived remediation for college were developed by 
considering several related variables: math and English results from EAP tests, SAT tests, AP 
coursework, and AP tests. The precise method for creating the composite outcome variables for 
math and English is described below (after the discussion of each component). 


Early Assessment Program (EAP) test score (Yo): In order to determine whether or not a 
student enters college without need for remediation, we used students’ early assessment program 
(EAP) scores as a proxy measure. The EAP test is a collaborative effort among the State Board 
of Education, the California Department of Education and the California State University (CSU) 
and is part of the California Standardized Tests (CST) administered to Grade 11 students (see 
McLean, 2012). The test was established to provide a measure of student readiness for college- 
level English and mathematics, and uses items from the California Standards Test (CST)-Math 
and CST-English in addition to 15 optional multiple choice questions. (It is important to point 
out that the additional 15 multiple choice questions are voluntarily taken by the student, after 
they have completed their CST tests, and students must indicate that CSU can receive their item 
responses. There is likely to be bias in who takes the exam as students who anticipate attending a 
CSU will be more motivated to take the exam.) Students who complete all test components are 
identified as either ready for CSU courses, conditionally ready, or not yet ready. Data for this 
outcome was requested directly from the OUSD Research department. We created an EAP- 
math variable and EAP-English variable where Grade 12 students who were deemed “ready for 
CSU courses” in math or “ready for CSU courses” in English receive a value of 1 for the 
appropriate variable, and Grade 12 students who were “conditionally ready” or “not yet ready” 
received a value of 0 for the appropriate variable. If a Grade 12 student had not taken either or 
both test, they received a value of 0 for the appropriate variable(s). 


SAT test score (Y%): Additionally as per the CSU criteria, students who score greater than a 550 
on the verbal and/or math sections of the SAT are identified as college-ready. Data for this 
outcome was requested directly from the OUSD Research department. We created an SAT- 
math variable and an SAT-English variable where Grade 12 students who scored greater than 
500 in either the math or verbal section of the SAT receive a value of 1 for the appropriate 
variable, and Grade 12 students who scored less than receive a value of 0 for the appropriate 
variable. (If students take the SAT more than once, we used the highest combined test score.) If 
a Grade 12 student had not taken either or both tests, they will receive a value of 0 for the 
appropriate variable(s). 


AP test scores (Y%): Students who scored greater than a 3 on an AP Calculus AB or BC, AP 
Statistics, and/or AP Language and Composition or Literature and Composition exams were 
identified as ready for college-level math and/or English. Data for this variable was requested 
directly from the OUSD Research department. We created an AP-exam-math variable and an 
AP-exam-English variable where Grade 12 students who scored greater than 3 on any of the 
indicated AP tests received a value of 1 for the appropriate variable, and Grade 12 students who 
did not score greater than 3 on any of the indicated tests received a value of 0 for the 
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appropriate variable. If a Grade 12 student had not taken any of these tests, they received a value 
of 0 for the appropriate variable(s). 


e AP coursework: Additionally as per the CSU criteria, students who in Grade 11 are deemed 
“conditionally ready” on the EAP Math test and go on to receive a C or better in Grade 12 in 
the following courses will be considered to be college ready in math: Pre-Calculus, honors or AP 
Physics, or successfully completing Algebra I for a second time. Data for this variable was 
requested directly from the OUSD Research department. We created an AP-course-math 
variable and an AP-course-English variable where Grade 12 students who received a C or better 
in any of the indicated AP courses received a value of 1 for the appropriate variable, and Grade 
12 students who did not receive a C or better in any of the indicated AP courses received a value 
of 0 for the appropriate variable. If a Grade 12 student had not taken any of the indicated tests, 
they received a value of 0 for the appropriate variable(s). 


We then calculated the final outcome composite variables (Waived for Remediation-Math and 
Waived for Remediation-English) in the following way. For each student, we added the variables for 
each component: EAP, SAT, AP-test, and AP-course separately for Math and English, to come up 
with a student score, ranging from 0 to 4. Any student with a value of 1 or greater was then recoded 
with a value of 1 to indicate “ready” for college math or college English; conversely, students with a 
value of 0 retained the value of 0 to indicate “not ready” for college math or college English 
accordingly. Then for each school, the percent of students ready for college math, and the percent 
of students ready for college English at each school was calculated, which is the number of Grade 12 
students ready for college math or college English divided by all Grade 12 CBED enrollments. 
Outcome comparisons were made between the percentage of grade 12 students who were college 
ready at the end of the baseline year (2011-12) and the percentage of grade 12 students who were 
college ready at the end of the final program year (2014-15) for all eight schools within OUSD. 


4. Outcomes: College enrollment rates (%) 

e College Enrollment. The National Student Clearinghouse (NSC) provides information on 
student enrollment status in two-/ four-year colleges and universities, public/private, trade, and 
vocational. This data is typically available in the school year following the first semester of 
enrollment at one of the colleges for whom there is a pre-agreement between NSC and the 
college. (For example, for a student who graduates in Spring 2012 (2011-12 school year), and 
then enrolls in college in Fall 2012 (2012-13 school year), their first semester data would not be 
available until Fall 2013 (2013-14.) Based on this data we documented whether or not a student 
enrolls in an institution of higher education, which is a more definittve outcome than a student’s 
intention to enroll and attend college. OUSD has a contractual relationship with NSC, and we 
requested this data directly from the OUSD Research department. For this outcome, we 
calculated the percent of Grade 12 students who subsequently enrolled in at least one semester 
of college in the following fall semester, as the number of Grade 12 students who enrolled in at 
least one semester divided by the Grade 12 CBEDS enrollments. Comparisons were made 
between the college enrollments of the Grade 12 students in the baseline year (2011-12) and the 
college enrollments of the grade 12 students in the third program year (2014-15) for all eight 
schools within OUSD. One potential limitation for this outcome measure was NSC uses social 
security numbers to match the list of OUSD students with students enrolled in two-/ four-year 
colleges, and we anticipate some under-reporting of data given the potentially large number of 
undocumented youth at these OUSD schools. However, we believe that this potential under- 
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reporting is consistent between the baseline and outcome measurement, and therefore will not 
result in a lessening of the possible impact. 


5. Independent Variables 

While the inclusion of additional covariates would conttol for between school differences, the 
current sample size (n=8) did not allow for this. Therefore, we did not include any additional 
covatiates. 


Statistical analysis of impacts on students 

Outcomes were compared between baseline year (2011-12) and the final program year (2014-15) for 
OUSD students at the targeted high schools. The comparison group for this pretest-posttest study 
includes grade 12 students who were enrolled in the eleven treatment schools during the school year 
prior to the introduction of the Oakland Accelerates process, i.e., the baseline year (2011-2012). 
Students at baseline had not been exposed to the intervention and therefore were considered the 
“pre-test” group. The analysis was conducted at the school level as OUSD students will not have 
both pre and post test scores for the outcome variables (e.g. an individual student cannot have a pre- 
test measure for graduation). At the school-level, a school has both a pre and posttest score for all 
outcome variables (e.g. a pre-test score for graduation rate and a post-test score for graduation rate). 


It is important to note that in 2012-13, six small learning community high schools from 2011-12 
were combined into two large high schools. Since the same students were combined into the same 
campuses, we combined schools in the 2011-12 sample (n=11) to represent the 2012-13 re- 
structuring (n=8) so that paired sample t-tests could be conducted within schools. 


Due to the nature of the study design — a one-group pretest-posttest at the school-level -- the 
evaluation sample and the study population are equivalent. Each outcome focuses on a different 
grade level, but each grade level sample includes all students in that grade, including under- 
represented students, African-American and Latino students. This design has several confounds 
including threats to internal validity associated with a quasi-experimental design (e.g. lack of a 
compatison group) and the N=1 problem (only one district is participating in the intervention). This 
means it is difficult to determine with any certainty whether change can be attributed to the 
intervention. Given the lack of statistical power, it is difficult to determine an effect with precision. 
Because the unit of assignment (district) is relatively large, it is not logistically or financially possible 
to recruit additional treatment or control districts. As a development study, we believe this design 
indicates whether the intervention shows evidence of promise. 


To address RQ2 and RQ3, five confirmatory contrasts were identified for the following four 
domains: 
Domain 1: High school graduation 

1. Graduation rates (%) 
Domain 2: College readiness 

2. A-g course completion rates (%) 
Domain 3: Waived remediation for college 

3. Proven proficiency in college-level math (%) 

4. Proven proficiency in college-level English (%) 
Domain 4: College enrollment 

5. First semester college enrollment rate (%) 
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The five outcomes noted above were each within their own domains, with the exception of 
outcomes 3 and 4, which were both within domain 3. For those outcomes which were in their own 
domain, no corrections for multiple comparisons were needed. Adjustments for multiple 
compatisons were needed for domain 3 since it contains two outcomes. We estimated the following 
paired sample t-tests to estimate intervention impacts for each outcome. 


yd 


[n 0") — Ord)’ 
n-1 


Whereby d= mean difference between baseline and posttest outcome variable; and 
n=number of schools. 


t= 


Another fifteen exploratory analyses were conducted to assess the impact of the intervention on 
underrepresented subgroups. A series of paired t-test identical to the one described above using 
school-level data were conducted separately to assess the effect of the Oakland Accelerates model 
on student college preparation for the three subgroups: African American students, Latino/a 
students and ELL students. For example, the analyses were conducted among a specific subgroup 
(e.g. all Latino) and paired t-tests were conducted for all relevant outcomes. In cases where the 
subgroup of interest was 10 students or fewer in a particular school, these schools were dropped 
from the analyses. 


Findings of Impact Study 2 


Table 11 demonstrates the findings of the paired sample t-tests on the ftve domains of college 
preparation for OUSD G12 students. These analyses indicate that the percentage of G12 students at 
each school who completed a-g course requirements, were ready for college English and/or were 
ready for college math did not significantly change (at the .05 level) from 2011-12 to 2014-15. The 
graduation rates — while they showed some improvement—also was not statistically significant. The 
one area that showed a statistical change was that of the percentage of G12 students at each school 
who enrolled in college in the semester after graduation. However, the change was for the worse 
with a statistically significant decrease at the .05 level (p=.047). Therefore, there is no evidence that 
the program had an impact on increasing college readiness or preparation for OUSD G12 students. 


Table 11. Summary of Unweighted Paired Sample T-test Analyses for OUSD Schools from Pretest to Posttest 


2011-12 2014-15 
N % SD N % SD t-statistic p-value 
% completed a-g course reqs 8 50.6 14.8 8 52.0 15.5 .322 757 
% ready for College English 8 25.6 15.5 8 21.7 15.2 -1.330 225 
% ready for College math 8 26.3 16.0 8 33.2 8.8 1.039 333 
Graduation rate 8 66.0 12.7 8 71.6 12.4 1.183 275 
% enrolling in college 8 62.8 13.8 8 51.8 3.6 -2.402 .047 


Exploratory Impact Analyses for Sub-groups 

Table 12-14 show the separate results of the exploratory subgroup analyses for African-American, 
Hispanic or ELL students. For the African-American subgroup, there was statistically significant 
decrease from 2011-12 to 2014-15 in the percentage of African-American students who enrolled in 
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college in the semester after high school graduation at the .05 level (p=.013). Moreover, there was a 
significant decrease in the percentage of African-American G12 students who were ready for 
college-level English at the .10 level (p=.067). For Hispanic and ELL subgroups, there were no 
statistically significant changes (at the .05 level) from 2011-12 to 2014-15 on the five domains of 
college preparation. Taken together, the findings indicate the Oakland Accelerates program had little 
effect on increasing college preparation for subgroups of African-American, Hispanic, and ELL G12 
students and, in fact, may have contributed negatively on African-American students’ readiness for 
college English and subsequent enrollment in college. 


Table 12. Summary of Unweighted Paired Sample T-test Analyses for African-American Students in OUSD 


Schools from Pretest to Posttest 


2011-12 2014-15 
N % SD N % sD t-statistic p-value 
% completed a-g course reqs 6 36.9 8.7 6 32.8 16.9 -.628 557 
% ready for college English 6 25.3 7.0 6 17.0 13.2 -2.413 061 
% ready for college math 6 12.9 7.3 6 21.2 14.0 1.737 .143 
Graduation rate 6 63.1 15.8 6 67.0 10.2 .611 .568 
% enrolling in college 6 65.0 10.9 6 44.6 14.1 -3.764 .013 


Table 13. Summary of Unweighted Paired Sample T-test Analyses for Hispanic Students in OUSD Schools from 


Pretest to Posttest 


2011-12 2014-15 
N % SD N % sD t-statistic p-value 
% completed a-g course reqs 7 49.7 14.5 7 51.0 15.7 722 497 
% ready for College English 7 19.3 12.8 7 15.6 8.2 -1.420 .205 
% ready for College math 7 20.3 14.4 7 27.3 8.5 1.023 346 
Graduation rate 7 63.1 10.7 7 69.5 14.3 1.308 .239 
% enrolling in college 7 54.0 11.3 7 46.8 8.6 -1.338 .229 


Table 14. Summary of Unweighted Paired Sample T-test Analyses for ELL Students in OUSD Schools from Pretest 


to Posttest 


2011-12 2014-15 
N % SD N % SD t-statistic p-value 
% completed a-g course reqs 5 22.8 6.1 5 25.3 9.3 465 666 
% ready for College English 5 5.7 6.9 5 5.8 2.7 035 974 
% ready for College math 5 7.4 2.6 5 12.8 7.5 1.670 .170 
Graduation rate? No data No data 
% enrolling in college 5 45.1 14.0 5 38.5 10.5 -1.796 147 


'S Graduation rates are not calculated by CDE for English language learners. 
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Conclusion 


In sum, these findings suggest that Oakland Accelerates may have had an impact on increasing the 
percentage of OUSD students taking the SAT exam, but it did not have a comparable impact on 
increasing the percentage of OUSD students taking the AP exams. Moreover, these findings suggest 
that Oakland Accelerates did not have the expected impact in terms of increasing performance on 
either the SAT or AP exams. In fact, it is possible that statistically increasing the percentage of 
OUSD students taking the SAT resulted in a zero net effect in changing the mean SAT combined 
score for OUSD students. 


In addition, there was no evidence that the program had an impact on increasing college 
preparation, such as completing a-g requirements, readiness for college English or college math, and 
the graduation rate, or actual college enrollment for OUSD G12 students. 


In exploratory subgroup analyses, we found evidence that Oakland Accelerates may have had an 
impact on the Hispanic subgroup increasing participation in AP tests from 2011-12 to 2014-15; but 
we found no evidence that the program had a similar impact for African-American or ELL 
subgroups. Moreover, so few students in the subgroups took the AP test that we were unable to 
assess the program impact on AP performance. Unfortunately we were unable to explore the impact 
of the program on subgroup SAT participation and performance, due to missing 2011-12 data for 
the comparison school district. 


Finally, we found little evidence that Oakland Accelerates had an impact on Hispanic, African- 
American or ELL subgroups’ college preparation — with the possible negative impact on decreasing 
the percentage of African-American students who were ready for college English or who enrolled in 
college after graduation. 


In retrospect, the program possibly had a greater impact on PSAT participation and AP course 
enrollment, as activities leading to these outcomes were actively pursued by Oakland Accelerates 
program staff and promoted by College Board leaders during the implementation (as documented in 
Oakland Accelerates implementation report). However, these particular outcomes were not 
identified as intended outcomes by the College Board when the logic model and research study was 
designed in 2011-12 and therefore, HTA did not register these confirmatory or exploratory contrasts 
for either impact study with the National Evaluation 13 Team. 
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Appendix: Registration Tables of Confirmatory & 
Exploratory Contrasts 
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Table A.1: Impact Study 1 Registration of Confirmatory Contrasts 


Contrast 
name 


Design 


c 
=s 
of 
= 2 
ee 
oo 
oa 


Age/grade 


intervention 


Exposure 


Comparison 
Group 


Condition; 
Description 


Outcome 


Measure 
[Scale]® 


observation 


Timing of 
measuremen 


Measure 
[Scale]® 


observation 


Timing of 
measuremen 


[Oakland [Business as 
Accelerates] usual (BAU)] Advanced 
QED 10th, 11% and 10%, 11, and = College Placement 
with 12 grade 12 grade Entrance and (AP) test 2011-12 AP 
AP test matche — students inthe students in 4 Placement performance End of the 2014- __ test End of the 
perfor d 8 treatment comparison Test (score) 2015 school performance 2011-2012 
mance schools _ group schools gt-12th  1-3years groupschools Achievement [Continuous] School _ year [Continuous] School _ school year 
[Oakland [BAU] 
Accelerates] gth 10th, 11th, Scholastic 
QED gth 10th, 41th, and 12th College Aptitude Test 
with and 12* grade grade Entrance and (SAT) 
SAT matche students in the students in 4 Placement performance End of the 2014- 2011-12 SAT End of the 
perfor d 8 treatment 1-3 comparison Test (mean score) 2015 school performance 2011-2012 
mance schools _ group schools gth_q2th — years group schools Achievement [Continuous] School year [Continuous] School — school year 
[Oakland 
Accelerates] [BAU] College 
QED 11th and 12th 11th and Entrance and Advanced 
with grade students 12th grade Placement Placement 2011-12 AP 
AP test matche  inthe8 students in4 Test (AP) test End of the 2014- participation End of the 
partici d treatment group 9th comparison participation participation 2015 school rate 2011-2012 
pation schools _ schools -12th 1-3 years groupschools [Continuous] (percentage) School year [Continuous] School — school year 
[Oakland College 
QED Accelerates] [BAU] Entrance and Scholastic 
with 12th grade 12th grade Placement Aptitude Test End of the 2014- 2011-12 SAT End of the 
SAT matche students in the studentsin4 Test (SAT) 2015 school participation 2011-2012 
partici d 8 treatment 10th comparison participation participation year rate school year 
pation schools _ group schools 12th 2-3 years groupschools [Continuous] (percentage) School [Continuous] School 


4 These names are provided for the AR Team’s administrative purposes only and you may choose whether or not to adopt them for your own use. 
5'The measurement scale describes how the measure is constructed. A measure may be categorized as continuous, ordinal, or binary. Please consult with your TA liaison if you have any 
questions regarding these measurement scales. 


¢The “unit of observation” is defined as the level at which the data are analyzed. For example, “Student” is listed if each student represents a single case in the dataset (as with individual 
level state test scores). “School” is listed if each school represents a single case in the dataset (as with school characteristics like AYP or school means of student test scores). 
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Table A.2: Impact Study 1 Registration of Exploratory Contrasts 


Contrast 
name 


AP test 
partici 
pation 


AP test 
partici 
pation 


AP test 
partici 
pation 


AP test 
perfor 
mance 


Design 


QED 
with 
matche 
d 
schools 


QED 
with 
matche 
d 
schools 


QED 
with 
matche 
d 
schools 


QED 
with 
matche 
d 
schools 


c 
=s 
of 
= 2 
ee 
oo 
oa 


[Oakland 
Accelerates] 
11th and 12th 
grade African 
American 
students in the 
8 treatment 
group schools 
[Oakland 
Accelerates] 
11th and 12th 
grade Hispanic 
students in the 
8 treatment 
group schools 
[Oakland 
Accelerates] 
11th and 12th 
grade ELL 
students in the 
8 treatment 
group schools 


[Oakland 
Accelerates] 
10th, 11% and 
12 grade 
African 
American 
students in 8 


treatment group 


schools 


intervention 


Age/grade 
Exposure 


gth 
-12th 1-3 years 
gth 
-12th 1-3 years 
gth 
-12th 1-3 years 
gth_q2th = 1-3 years 


Comparison 
Group 


Condition; 
Description 


[BAU] 

11th and 
12th grade 
African 
American 
students in 4 
comparison 
group schools 
[BAU] 

11th and 
12th grade 
Hispanic 
students in 4 
comparison 
group schools 
[BAU] 

11th and 
12th grade 
ELL students 
in4 
comparison 
group schools 


[Business as 
usual (BAU)] 
10, 11% and 
12¢* grade 
African 
American 
students in 4 
comparison 
group schools 


College 
Entrance and 
Placement 
Test 
participation 
[Continuous] 


College 
Entrance and 
Placement 
Test 
participation 
[Continuous] 


College 
Entrance and 
Placement 
Test 
participation 
[Continuous] 


College 
Entrance and 
Placement 
Test 
Achievement 


Outcome 


Measure 
[Scale] 


African 
American 
Advanced 
Placement 
(AP) test 
participation 
(percentage) 


Hispanic 
Advanced 
Placement 
(AP) test 
participation 
(percentage) 


ELL 
Advanced 
Placement 
(AP) test 
participation 
(percentage) 


African 
American 
Advanced 
Placement 
(AP) test 
performance 
(score) 
[Continuous] 


observation 


School 


School 


School 


School 


Timing of 
measuremen 


End of the 2014- 
2015 school 
year 


End of the 2014- 
2015 school 
year 


End of the 2014- 
2015 school 
year 


End of the 2014- 
2015 school 
year 


Measure 
[Scale]® 


African 
American 
2011-12 AP 
participation 
rate 
[Continuous] 


Hispanic 2011- 
12 AP 
participation 
rate 
[Continuous] 


ELL 2011-12 
AP 
participation 
rate 
[Continuous] 


African 
American 
2011-12 AP 
test 
performance 
[Continuous] 


observation 
Timing of 
measuremen 


End of the 
2011-2012 


School — school year 


End of the 
2011-2012 


School — school year 


End of the 
2011-2012 


School _ school year 


End of the 
2011-2012 


School — school year 
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Contrast 
name 


AP test 
perfor 
mance 


AP test 
perfor 
mance 


Design 


QED 
with 
matche 
d 
schools 


QED 
with 
matche 
d 
schools 


c 
es 
ow 
s 2 
ee 
oo 
oa 


[Oakland 
Accelerates] 
10th, 11 and 
12h grade 
Hispanic 
students in 8 
treatment group 
schools 
[Oakland 
Accelerates] 
10%, 11th and 
12" grade ELL 
students in 8 
treatment group 
schools 


Age/grade 


gth.4 2th 


gth_4 th 


intervention 


Exposure 


1-3 years 


1-3 years 


Comparison 
Group 


Condition; 
Description 


[Business as 
usual (BAU)] 
10th, 11% and 
12 grade 
Hispanic 
students in 4 
comparison 
group schools 
[Business as 
usual (BAU)] 
10%, 11th and 
12" grade ELL 
students in 4 
comparison 
group schools 


College 
Entrance and 
Placement 
Test 
Achievement 


College 
Entrance and 
Placement 
Test 
Achievement 


Outcome 


Measure 
[Scale]® 
observation 


Hispanic 
Advanced 
Placement 
(AP) test 
performance 
(score) 
[Continuous] 
ELL 
Advanced 
Placement 
(AP) test 
performance 
(score) 
[Continuous] 


School 


School 


Timing of 
measuremen 


End of the 2014- 
2015 school 
year 


End of the 2014- 
2015 school 
year 


Measure 
[Scale]® 
observation 


Hispanic 2011- 
12 AP test 
performance 


[Continuous] School 


ELL 2011-12 
AP test 
performance 


[Continuous] School 


4 These names are provided for the AR Team’s administrative purposes only and you may choose whether or not to adopt them for your own use. 
>The measurement scale describes how the measure is constructed. A measure may be categorized as continuous, ordinal, or binary. Please consult with your TA liaison if you have any 
questions regarding these measurement scales. 
©The “unit of observation” is defined as the level at which the data are analyzed. For example, “Student” is listed if each student represents a single case in the dataset (as with individual 


level state test scores). “School” is listed if each school represents a single case in the dataset (as with school characteristics like AYP or school means of student test scores). 


Timing of 
measuremen 


End of the 
2011-2012 
school year 


End of the 
2011-2012 
school year 
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Table A.3: Impact Study 2 Registration of Confirmatory Contrasts 


o 

£ 

o 

< 

a 

© 

a] 

(=| 

° 

s) 
High School Pre- 
Graduation post 
College Pre- 
Enrollment post 
Waived 
Remediation 
in College Pre- 
English post 
Waived 
Remediation 
in College Pre- 
Math post 
College Pre- 
Readiness post 


Design 


Treatment Group 


< 
aS} 
> 
2 
= 
1s} 
7) 
o 
a 


12 grade 
students attending 
OUSD in the final 
year of the study 
(SY2014-15) 

12¢* grade 
students attending 
OUSD in the final 
year of the study 
(SY2014-15) 

12th grade 
students attending 
OUSD in the final 
year of the study 
(SY2014-15) 

12h grade 
students attending 
OUSD in the final 
year of the study 
(SY2014-15) 

12 grade 
students attending 
OUSD in the final 
year of the study 
(SY2014-15) 


Age/grade 


10th. 
42th 
grade 


10th = 
42th 
grade 


10th. 
12th 
grade 


10th. 
12th 
grade 


10th. 
12th 
grade 


intervention 


Exposure 


years 


years 


years 


years 


3 
years 


Comparison 
Group 


Description 


12¢ grade 
students 
attending OUSD 
at study baseline 
(SY2011-12) 

12th grade 
students 
attending OUSD 
at study baseline 
(SY2011-12) 

12'h grade 
students 
attending OUSD 
at study baseline 
(SY2011-12) 

12¢ grade 
students 
attending OUSD 
at study baseline 
(SY2011-12) 

12¢ grade 
students 
attending OUSD 
at study baseline 
(SY2011-12) 


High School 
Graduation 


College 
Enrollment 


Waived 
Remediatio 
nin College 
English 


Waived 
Remediatio 
nin College 
Math 


College 
Readiness 


Outcome (Post-test) 


Measure 
[Scale]° 


Graduation 
Rate 
[Continuous] 


College 
Enrollment 
[Continuous] 
Passing rate for 
composite of 
EAP, AP, and 
SAT (English) 
[Continuous] 
Passing rate for 
composite of 
EAP, AP, and 
SAT (math) 
[Continuous] 


A-g course 
completion 
[Continuous] 


observation 


School 


School 


School 


School 


School 


@ These names are provided for the AR Team’s administrative purposes only and you may choose whether or not to adopt them for your own use. 
>’The measurement scale describes how the measure is constructed. A measure may be categorized as continuous, ordinal, or binary. Please consult with your TA liaison if you have any questions regarding these 


measurement scales. 


Timing of 
measurement 


At end of 
school 
year 
2014-15 


At end of 
school 
year 
2014-15 


At end of 
school 
year 
2014-15 


At end of 
school 
year 
2014-15 


At end of 
school 
year 
2014-15 


Outcome (Pre-test) 


Measure 
[Scale]° 


Graduation rate 
[Continuous] 


College 
Enrollment 
[Continuous] 
Passing rate for 
composite of 
EAP, AP, and 
SAT (English) 
[Continuous] 
Passing rate for 
composite of 
EAP, AP, and 
SAT (math) 
[Continuous] 


A-g course 
completion 
[Continuous] 


observation 


School 


School 


School 


School 


School 


measurement 


Fall of 2012 
(after end of 
school year 
2011-12) 
After end of 
12th grade 


Fall of 2012 
(after end of 
school year 
2011-12) 


Fall of 2012 
(after end of 
school year 
2011-12) 


Fall of 2012 
(after end of 
school year 
2011-12) 


Fall of 2012 
(after end of 
school year 
2011-12) 


©The “unit of observation” is defined as the level at which the data are analyzed. For example, “Student” is listed if each student represents a single case in the dataset (as with individual level state test scores). 


“School” is listed if each school represents a single case in the dataset (as with school characteristics like AYP or school means of student test scores). 
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Table A.4: Impact Study 2 Registration of Exploratory Contrasts 


Contrast name? 


High School 
Graduation 


High School 
Graduation 


High School 
Graduation 


College 
Enrollment 
College 
Enrollment 


Design 


Pre-post 


Pre-post 


Pre-post 


Pre-post 


Pre-post 


Treatment Group 


< 
& 
> 
2 
= 
oO 
nn 
cf 
a 


12th grade 
African- 
American 
students 
attending OUSD 
in the final year 
of the study 
(SY2014-15) 


12th grade 
Latino students 
attending OUSD 
in the final year 
of the study 
(SY2014-15) 


12 grade ELL 
students 
attending OUSD 
in the final year 
of the study 
(SY2014-15) 
12* grade 
African- 
American 
students 
attending OUSD 
in the final year 
of the study 
(SY2014-15) 
12 grade 
Latino students 


Age/grade 
during 


10¢- 
12th 
grade 


10th- 
12th 
grade 


10¢- 
42th 
grade 


10th = 
12th 
grade 
10% - 
42th 


intervention 


Exposure 


3 
years 


years 


years 


Comparison 
Group 


Description 


12 grade 
African- 
American 
students 
attending 
OUSD at study 
baseline 
(SY2011-12) 


12 grade 
Latino students 
attending 
OUSD at study 
baseline 
(SY2011-12) 


12 grade ELL 
students 
attending 
OUSD at study 
baseline 
(SY2011-12) 
12* grade 
African- 
American 
students 
attending 
OUSD at study 
baseline 
(SY2011-12) 
12 grade 
Latino students 


High School 
Graduation 


High School 
Graduation 


High School 
Graduation 


College 
Enrollment 
College 
Enrollment 


Outcome (Post-test) 


Measure 
[Scale]® 


Graduation 
Rate among 
African- 
American 
students 
[Continuous] 


Graduation 
Rate among 
Latino students 
[Continuous] 


Graduation 
Rate among ELL 
students 
[Continuous] 


College 
Enrollment 
among African- 
American 
students 
[Continuous] 
College 
Enrollment 


observation‘ 


School 


School 


School 


School 


School 


Timing of 
measurement 


At end of 
school 
year 
2014-15 
After end 
of 12th 
grade 

At end of 
school 
year 
2014-15 
After end 
of 12th 
grade 

At end of 
school 
year 
2014-15 
After end 
of 12% 
grade 


At end of 
school 
year 
2014-15 
At end of 
school 


Outcome (Pre-test) 


Measure 
[Scale] 


Graduation rate 
among African- 
American 
students 
[Continuous] 


Graduation rate 
among Latino 
students 
[Continuous] 


Graduation rate 
among ELL 
students 
[Continuous] 


College 
Enrollment 
among African- 
American 
students 
[Continuous] 
College 
Enrollment 


observation‘ 


School 


School 


School 


School 


School 


Timing of 
measurement 


Fall of 2012 
(after end of 
school year 
2011-12) 
After end of 
12" grade 


Fall of 2012 
(after end of 
school year 
2011-12) 
After end of 
12th grade 


Fall of 2012 
(after end of 
school year 
2011-12) 
After end of 
12" grade 


Fall of 2012 
(after end of 
school year 
2011-12) 
Fall of 2012 
(after end of 
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attending OUSD _ grade attending among Latino year among Latino school year 
in the final year OUSD at study students 2014-15 students 2011-12) 
of the study baseline [Continuous] [Continuous] 
(SY2014-15) (SY2011-12) 
12th grade ELL 12th grade ELL 
students students College College 
attending OUSD attending Enrollment At end of Enrollment Fall of 2012 
in the final year 10¢ - OUSD at study among ELL school among ELL (after end of 
College of the study 12th 3 baseline College students year students school year 
Enrollment Pre-post (SY2014-15) grade years (SY2011-12) Enrollment [Continuous] School 2014-15 [Continuous] School 2011-12) 
Passing rate for Passing rate for 
12th grade 12 grade composite of composite of 
African- African- EAP, AP, and EAP, AP, and 
American American SAT among SAT among 
students students African- African- 
Waived attending OUSD attending Waived American At end of American Fall of 2012 
Remediation in the final year 10¢- OUSD at study Remediatio students school students (after end of 
in College of the study 12th 3 baseline nin College (English) year (English) school year 
English Pre-post (SY2014-15) grade years (SY2011-12) English [Continuous] School 2014-15 [Continuous] School 2011-12) 
Passing rate for Passing rate for 
12th grade 12 grade composite of composite of 
Latino students Latino students EAP, AP, and EAP, AP, and 
Waived attending OUSD attending Waived SAT among At end of SAT among Fall of 2012 
Remediation in the final year 10th. OUSD at study Remediatio —_ Latino students school Latino students (after end of 
in College of the study 12th 3 baseline nin College (English) year (English) school year 
English Pre-post (SY2014-15) grade years (SY2011-12) English [Continuous] School 2014-15 [Continuous] School 2011-12) 
Passing rate for Passing rate for 
12" grade ELL 12" grade ELL composite of composite of 
students students EAP, AP, and EAP, AP, and 
Waived attending OUSD attending Waived SAT among ELL At end of SAT among ELL Fall of 2012 
Remediation in the final year 10¢- OUSD at study Remediatio students school students (after end of 
in College of the study 12th 3 baseline nin College (English) year (English) school year 
English Pre-post (SY2014-15) grade years (SY2011-12) English [Continuous] School 2014-15 [Continuous] School 2011-12) 
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Contrast name? 


Waived 
Remediation 
in College 
Math 


Waived 
Remediation 
in College 
Math 


Waived 
Remediation 
in College 
Math 


College 
Readiness 
College 
Readiness 


Design 


Pre-post 


Pre-post 


Pre-post 


Pre-post 


Pre-post 


Treatment Group 


< 
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= 
2 
= 
oO 
vn 
o 
a 


12th grade 
African- 
American 
students 
attending OUSD 
in the final year 
of the study 
(SY2014-15) 


12 grade 
Latino students 
attending OUSD 
in the final year 
of the study 
(SY2014-15) 


12th grade ELL 
students 
attending OUSD 
in the final year 
of the study 
(SY2014-15) 
12* grade 
African- 
American 
students 
attending OUSD 
in the final year 
of the study 
(SY2014-15) 
12 grade 
Latino students 


Age/grade 


10t- 
12th 
grade 


10th. 
12th 
grade 


10¢- 
12th 
grade 


10th- 
12th 
grade 
10¢- 
12th 


intervention 


Exposure 


years 


years 


years 


Comparison 
Group 


Description 


12 grade 
African- 
American 
students 
attending 
OUSD at study 
baseline 
(SY2011-12) 


12¢* grade 
Latino students 
attending 
OUSD at study 
baseline 
(SY2011-12) 


12th grade ELL 
students 
attending 
OUSD at study 
baseline 
(SY2011-12) 
12* grade 
African- 
American 
students 
attending 
OUSD at study 
baseline 
(SY2011-12) 
12th grade 
Latino students 


Waived 
Remediatio 
nin College 
Math 


Waived 
Remediatio 
nin College 
Math 


Waived 
Remediatio 
nin College 
Math 


College 
Readiness 
College 
Readiness 


Outcome (Post-test) 


[Scale]° 


Passing rate for 
composite of 
EAP, AP, and 
SAT among 
African- 
American 
students 
(math) 
[Continuous] 
Passing rate for 
composite of 
EAP, AP, and 
SAT among 
Latino students 
(math) 
[Continuous] 
Passing rate for 
composite of 
EAP, AP, and 
SAT among ELL 
students 
(math) 
[Continuous] 


A-g course 
completion 
[Continuous] 
A-g course 
completion 


observation’ 


School 


School 


School 


School 


School 


Timing of 
measurement 


At end of 
school 
year 
2014-15 


At end of 
school 
year 
2014-15 


At end of 
school 
year 
2014-15 


At end of 
school 
year 
2014-15 
At end of 
school 


Outcome (Pre-test) 


Measure 
[Scale] 


Passing rate for 
composite of 
EAP, AP, and 
SAT among 
African- 
American 
students (math) 
[Continuous] 
Passing rate for 
composite of 
EAP, AP, and 
SAT among 
Latino students 
(math) 
[Continuous] 


Passing rate for 
composite of 
EAP, AP, and 
SAT among ELL 
students (math) 
[Continuous] 


A-g course 
completion 
among African- 
American 
students 
[Continuous] 
A-g course 
completion 


observation‘ 


School 


School 


School 


School 


School 


Timing of 
measurement 


Fall of 2012 
(after end of 
school year 
2011-12) 


Fall of 2012 
(after end of 
school year 
2011-12) 


Fall of 2012 
(after end of 
school year 
2011-12) 


Fall of 2012 
(after end of 
school year 
2011-12) 
Fall of 2012 
(after end of 
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attending OUSD _ grade attending among Latino year among Latino school year 
in the final year OUSD at study students 2014-15 students 2011-12) 
of the study baseline [Continuous] [Continuous] 
(SY2014-15) (SY2011-12) 
12th grade ELL 12th grade ELL 
students students A-g course A-g course 
attending OUSD attending completion At end of completion Fall of 2012 
in the final year 10¢- OUSD at study among ELL school among ELL (after end of 
College of the study 12th 3 baseline College students year students school year 
Readiness Pre-post (SY2014-15) grade years (SY2011-12) Readiness [Continuous] School 2014-15 [Continuous] School 2011-12) 


@ These names are provided for the AR Team’s administrative purposes only and you may choose whether or not to adopt them for your own use. 

>’The measurement scale describes how the measure is constructed. A measure may be categorized as continuous, ordinal, or binary. Please consult with your TA liaison if you have any questions regarding these 
measurement scales. 

©The “unit of observation” is defined as the level at which the data are analyzed. For example, “Student” is listed if each student represents a single case in the dataset (as with individual level state test scores). 
“School” is listed if each school represents a single case in the dataset (as with school characteristics like AYP or school means of student test scores). 
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Background 


Oakland Unified School District (QUSD) has been developing and implementing Oakland 
Accelerates since January 2012 to improve college readiness among OUSD students. Driven by 
poor student college readiness outcomes and faced with isolated efforts of individual counselors, 
teachers, principals, or discrete extended-day programs, OUSD administrators were seeking to 
mount a concerted effort by a// educators (teachers, site and district leaders, and counselors) to 
embed college readiness expectations and supports across the entire school experience. The College 
Board brought experience collaborating with educators in individual schools to develop such a 
coordinated effort, but it had not yet conducted such a process with educators across an entire 
school district and especially one as large and with as many challenges as OUSD. 


OUSD was inspired to apply for a prestigious four-year, $3 million' Department of Education 
Investing in Innovations (i3) Development grant. This grant would allow OUSD to implement 
The College Board’s EXCELerator program across the district and to create an internal cadre of 
trained college readiness specialists to support district-level improvement in classroom rigor and 
college-going culture. The ambitious goals were as follows: 

e Goal 1: Increase student SAT/AP exam participation and performance 

© Goal 2: Improve student college readiness 


e Goal 3: Increase readiness for college-level math and English coursework 
e Goal 4: Build a college-going culture at OUSD high schools 


An initial strategic work plan for Oakland Accelerates activities was driven by a “District 
Diagnostic” planning process whereby The College Board’s EXCELerator team conducted a gap 
analysis with high-level OUSD administrators from November 2011 through June 2012. While there 
were many areas for improvement identified during the process, the District Diagnostic strategic 
work plan outlined the following three priorities for improving the OUSD college readiness system: 
e Priority 1: Create a college & career district infrastructure supporting common policies, practices and 
processes that establish college readiness as a priority at every level of the organization 
e = Priority 2: Develop a rigorous curticulum and instruction for all students that is vertically aligned and 
includes key content knowledge and cognitive strategies for college readiness 
e Priority 3: Build a data-driven accountability system that identifies readiness indicators, monitors 
progtess, and supports each high school in achieving its college and career readiness goals 


Besides describing how OUSD would create a “college readiness team,” neither the grant application 
nor the strategic work plan specified exactly how the identified priorities might be addressed. For 
this reason, many Oakland Accelerates activities were developed on an ad hoc basis via the 
collaborative partnership between OUSD implementation team members (a combination of 
administrators and teachers) and The College Board’s EXCELerator implementation team between 
July 2012 and June 2014. Since June 2014, the OUSD implementation team members continued to 
develop and incorporate new components that reflect the initial mission of the Oakland Accelerates 
program. 


' With required matching funds of $527, 000 
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Summary of Evaluation Design 


Beginning in January 2012, Oakland Unified School District contracted with Hatchuel Tabernik 
and Associates (HTA) to conduct an independent evaluation of the i3 grant-funded Oakland 
Accelerates (2011 cohort). The primary goal of the evaluation is to understand the program’s impact 
on student academic performance outcomes and on changing adult perceptions towards building a 
college-going culture for all students at OUSD high schools. The evaluation also serves to document 
what worked and did not work, the challenges faced and lessons learned, as well as successes 
achieved. In accordance with Department of Education 13 grant requirements, the Oakland 
Accelerates evaluation plan* was formally reviewed and approved through a peer-review process. 
(See Table 1 for an overview of the evaluation design.) In addition, HTA received one-on-one 
assistance from Abt Associates TA provider.” 


AVENUE yale MCler-]| 


Increase student SAT/AP 
exam participation & 
performance 


Improve student college 
readiness 


Reduce need for remedial 
coursework in math and 


English 


Build a college-going culture 
F) 0) Uh) 


Understand what worked, 
challenges, and lessons 
FY al=te| 


Independence of evaluation 


Research Design 


Nonequivalent comparison 
group design at school-level 


Table 1: Summary of Evaluation Design in Oakland Accelerates 2012-13 Evaluation Plan 


Data Sources 


Student data from OUSD & 
comparison school district 


One-group pretest-posttest 
design at school level 


Student data from OUSD 


Mixed methods one-group 
pretest-posttest design 


Pre-/post-faculty & 
administrator surveys; and 
annual key stakeholder 
interviews 


Fidelity to implementation 
qualitative design 


Program documents and logic 
models; and fidelity matrix 
dashboards* 


The evaluation is being conducted to meet the standards for independence. Specifically, data 
collection for outcomes, with the exception of achievement tests administered by the district, state, 
or nationally, and the analysis of outcome data is conducted independently. Furthermore, findings 
reported to the National Evaluation 13 (NEi3) team will not be subject to the approval of the project 
developer/grantee (i.e., College Board & OUSD). 


? Acopy of the approved Oakland Accelerates 2013 evaluation plan can be requested from HTA. 


3 See http://ies.ed.gov/ncee 


rojects/evaluation/assistance_ita.as 


for more details on the i3 national evaluation. 


* HTA created a fidelity matrix dashboard for College Board to track key implementation activities on a quarterly 
basis. A copy of the matrix is in the Appendix. 
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About the Implementation Report 

This implementation report summarizes formative data (primarily qualitative) collected from January 
2012 through May 2015. Baseline student achievement data has already been collected, and follow- 
up data will be collected between May and October 2015, for the 2014-15 school year. The final 
summative impact report will include data analyses of final student outcome data and results from 
faculty and administrator surveys. The summative data analyses will be reflected in the final 
comprehensive impact report and delivered to the Department of Education and OUSD between 
March and July 2016. 


The table below summarizes the data sources used in the evaluation, noting which elements are in 
the implementation report and which will be added into the final impact report. 


Table 2: Data Presented in Implementation and Final Summative Evaluation Reports 


Data Source Implementation Final Impact Report 
Report 


Key stakeholder interviews 


2012-2015 
Program documents & logic models 3012-15 
Fidelity matrix dashboards 9012-14 
Student achievement data 9011-14 +) 2014-15 


Faculty & administrator surveys 


+) 2012-13; 2014-15 


Note: All names have been changed to initials to protect their privacy. 
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Summary of Findings 
Major Accomplishments/Successes 


9" grade college and career community plans. With a goal of 100% completion of CCPs, the 
team has increased the number of 9" grade students with a plan every year since the beginning of the 
grant. 

District-wide institutionalization of the PSAT for all 10" grade students. With all 10% graders 
taking the PSAT, students gain early SAT test-taking skills and students and teachers gain access to 
rich data to better assess students’ strengths and weaknesses in preparation for college and college- 
ready classes. The policy also conveys a powerful message from the district: that all students are 
capable of going to college. 

Improved communication and coordination for the AP Teacher Professional Development 
trainings. Teacher attendance increased since the first year of the program and participants reported 
that the trainings very beneficial and that they instituted what they learned immediately in their 
classrooms. 

Increased collaboration, communication and coalition building among program staff and district 
players working towards similar goals by the third year of the program. Administrative shifts in how 
the program was managed meant more cross team work and that efforts could be more efficiently 
streamlined and unduplicated. Interviewees mentioned how departments are working together and 
program team members have developed a true collaborative partnership by understanding and 
valuing the strengths each individual brings. 

Looking towards sustainability, CR and Content Specialists have put successful systems in place for 
designing and implementing professional learning in schools. Interviewees expressed that with the 
support of College Board-supplied experts and program collaboration, Specialists have created 
systems that can continue sustainably into the future. Examples include data-driven modules for 
teachers, conversations around college transcripts, and processes and calendars for preparing schools 
for the aforementioned annual PSAT and College Career and Community Plans. 


Challenges 


Several major contractual delays between OUSD and the College Board created 
implementation challenges including disconnecting the progress made with the District 
Diagnostic needs assessment, requiting much time in the 2012-13 school year spent 
revisiting concepts, adding new information, and taking new directions in team planning 
rather than implementing a stable program. 

Program details describing the elements to be implemented were not clearly defined. As a result, 
many Oakland Accelerates activities were developed on an ad hoc basis via the collaborative 
partnership between OUSD implementation team members and The College Board’s 
implementation team between July 2012 through June 2014. 

Difference in opinions among members of the partnership around strategies and actions 
proposed in the grant application versus what they personally believed needed to be done to address 
the issues. Perhaps in part because of the lack of agreement and buy-in as well as the delay in 
assigning or hiring key staff, the Oakland Accelerates Project Director considered the Year 1 Work 
Plan produced by College Board to be open to adjustments and modifications, and in the APR, 
referred to the Fall 2012 term to be the actual “start-up phase.” 
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e Differences in opinions on scope and breadth of program. One major area of disagreement was 
whether College Board’s SpringBoard program, which was already occurring in the middle schools, 
should be considered part of the Oakland Accelerates program strategy as well. College Board 
EXCELerator team members, in particular, believed this work was vital to the grant’s goals. 
However, some of the OUSD leadership disagreed — stating that the grant was meant to focus on AP 
courses in high school, not pre-AP courses in middle school. 

e Differences in opinion on roles and responsibilities. Another area of discourse centered on the 
role and responsibilities of the College Readiness (CR) Specialists and the Implementation Manager. 
Some argued that the CR Specialists needed to be focused on rigor as content specialists and coaches 
to AP teachers, whereas others believed the CR Specialists should be focused on college 
preparedness more generally, working with principals and counselors. As a College Board employee, 
the Implementation Manager was directed to do a set of tasks that the outgoing EXCELerator team 
members as well as the grant had outlined for her. However, being part of a partnership with OUSD 
and working in an unfamiliar school district, she had to be flexible and adapt to what OUSD team 
members wanted. As a result, a number of discussions in early 2013 centered on what she should or 
should not be doing as part of the intervention. 

e Transitions and Turnover. At College Board, a new President and CEO was hired in the fall of 
2012, and within a few months, the entire EXCELerator project was scrapped. The five-person 
College Board team was reduced to a new Implementation Manager who was new to College Board 
as well as to OUSD, as the only full-time College Board representative. In April 2013 at OUSD, 
Superintendent T\S., a key partner in the project with a strong relationship with College Board, 
announced his departure effective in June. Later in the school year, the High School Network 
Director, another key partner in the project, announced her retirement effective July 2013. 
Consequently, the project team lost the members in high-level leadership positions at OUSD to 
maintain the visibility of the work. As of July 2014, College Board ended its participation in the 
Oakland Accelerates project, arguably six months earlier than was originally proposed in the grant. 
The Implementation Manager was laid off by College Board after June 2014, and any College Board 
support ended along with her departure. 

e Lack of participation from district leadership. Without the direct line to district leadership as 
existed in 2012-13 with Superintendent T.S. and the High School Network Officer, the project staff 
struggled with communication and coordination between departments and offices. In the opinion of 
some key stakeholders, many competing initiatives operated in parallel with Oakland Accelerates 
activities, often utilizing Oakland Accelerates staff, which made it difficult to establish priority for 
many program activities called out in the Strategic Work plan. 


Figure 1 below displays the timeline of major milestones and program activities 
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Figure 1: Major Milestones during Oakland Accelerates Implementation and Evaluation, 2012-2015 


CB RETIRES EXC, 


6 CR SPECIALISTS PULLS 
ASSIGNED RESOURCES 
C.P. STARTS AS (EXCEPT M.H.) 
OA PROJ DIR M.H. STARTS AS 
EXC IMPL MGR 
CB STARTS OA 
PLANNING WITH MEME CLAN SUP T.S. RETIRES SUP A.W. BEGINS 
EXEC LEADERS 
APPROVED 
DISTRICT 
DIAGNOSTIC 
CONDUCTED 


© © © © © 


Nov 2011 Feb 2011 Jul 2012 Sept 2012 Nov 2012 Dec 2012 Dec2012 Jun 2013 Jul 2013 Sept 2013 June 2014 July 2014 July 2015 Dec 2015 


G.L. NAMED NEW A.A. NAMED NEW OA 
OA PROJ DIR ses 
| 
5 OF 6 CR LEAVE 13 GRANT ENDS 


CB DEPARTS OA 


Legend: 

CB= College Board 
OA=Oakland Accelerates 

CR= College Readiness Specialist 
EXC= EXCELerator 
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Start-Up Phase: January-June 2012 


Table 1. Start-Up Team 


Name Title/Position Organization Funding Source 
TS. Superintendent OUSD OUSD 

MSS. Deputy Superintendent OUSD OUSD 

AM. Exec. Officer, High Schools OUSD OUSD 

M.D Exec. Officer, School Transformation OUSD OUSD 

JJ. Director, College/Career Readiness OUSD OUSD, Irvine Grant 
C.P. Project Director, SLCs OUSD OUSD, SLC Grant 
Not yet hired Project Director, Oakland Accelerates OUSD 13 Grant 

Not yet hired CR Spec. 1 OUSD 13 Grant 

Not yet hired CR Spec. 2 OUSD 13 Grant 

B.K. Sr. Director, College Readiness Initiative | College Board College Board 

Not yet hired Project Manager, EXCELerator College Board 13 Grant 

L.B. Director, Implementation College Board College Board 
L.G. Exec. Director, Implementation College Board College Board 
L.D. Sr. District Director, Western Region College Board College Board 
DW. St. Director, Research & Analytics College Board College Board 

lel Evaluator HTA 13 Grant 


Proposed Program Activities 


1. Review District Diagnostic findings and recommendations 


Develop Oakland Accelerates Strategic Plan 


Hire full-time EXCELerator Project Manager 


Administer the EPIC CollegeCareerReady online school diagnostic 


Establish roles and responsibilities of College Readiness Specialists 


Define Oakland Accelerates/EXCELerator Partnership (roles & responsibilities) 


Hire College Readiness (CR) Specialists & begin the work of the CR Specialist team 


Cn aS FS LS 


Create evaluation and research design in accordance with Education Department requirements and 
begin evaluation 


Adherence to Proposed Program Activities 


1. District Diagnostic 
From February through April 2012, Senior Director of EXCELerator Implementation at College 
Board, B.K. (see Table 1), took the lead in presenting the findings of the District Diagnostic to 
OUSD district leaders. The College Board had already completed the District Diagnostic by 
October 2011 (before the 13 award was granted) as part of the Castlemont Corridor partnership 
between College Board and OUSD. The District Diagnostic was a gap analysis of five components 
that College Board believed were critical to successful college readiness in a school district. A 
summary score was compiled according to a multidimensional rubric: 
1) District infrastructure which measures the degree to which district goals, policies, and processes 
establish college readiness as a priority (Rated as Phase 1 which is the lowest score on the District 
Diagnostic); 
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2) Curriculum coherence which assesses whether district course guidelines are aligned with college 
readiness (Phase 1); 

3) Assessments that inform and are used to guide students on a path to college readiness (Phase 2); 

4) Student academic support (Phase 1); and 

5) Culture and community® (Phase 2). 


Overall, OUSD earned a Phase 1 “General Awareness” rating based on this rubric, with the lowest 
rating being Phase 1 and the highest rating being Phase 4 “System-wide Integration.” In the 
Diagnostic Final Report, College Board emphasized three major priority areas to improve college 
readiness in OUSD: 
(a) Build a college and career readiness district infrastructure supporting common policies, practices 
and processes that establish college readiness as a priority at every level of the organization; 
(b) Create a rigorous curriculum and instruction for all students that is vertically aligned and includes 
key content knowledge and cognitive strategies for college readiness; and 
(c) Develop a data-driven accountability system that identifies readiness indicators and monitors 
progtess, supports each high school in achieving its college and career readiness goals, and specifies a 


two-way communication strategy between district and school leaders. 


There were also specific priority areas identified for each of the five components in the District 
Diagnostic, but these were general strategies, not specific action plan items (e.g., with reference to 
building district infrastructure, it was suggested that OUSD should “develop a process for 
identifying and generalizing college readiness related research and best practices.’’) 


From February through April 2012, as part of the start-up phase activities, B.K. presented the 
findings of this Diagnostic to OUSD leadership and began to develop the Oakland Accelerates 
Strategic Plan. B.K. engaged in: five strategy/planning meetings; four meetings with the Department 
of Leadership, Curriculum, and Instruction; one meeting with the Career and College Readiness 
Office; and one meeting with all executive OUSD leadership (including Superintendent T.S., Deputy 
Superintendent MLS. and all other executive-level leaders). 


2. EPIC CollegeCareerReady School Diagnostic 

Although stated in the grant, the completion of this task was later deemed unnecessary. According 
to meeting minutes, the EPIC School Diagnostic was administered to six schools with “varying 
success.” (It is not clear whether these were six high schools, or a combination of high schools and 
middle schools.) However in a May 2012 meeting, College Board clarified that the EPIC School 
Diagnostic was not required for successful implementation of the EXCELerator model. 


3. Oakland Accelerates Strategic Plan 

By June 6, 2012, B.K. had created an initial 2012-13 Project Management Plan Summary and shared 
it with the initial EXCELerator/Oakland Accelerates Partnership team. This provided a little more 
detail regarding the three priority areas, as follows: 


a. Build(ing) a district college and career readiness district infrastructure was specifically linked to 
developing job descriptions for and hiring College Readiness (CR) Specialists who would: 


> College Board modified the District Diagnostic to change the 5 component from “Student Family Support” to 
“Culture and Community,” specifically for OUSD. 
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a. Work with eight high schools and seven middle schools to develop various aspects of college 
readiness; 

Support the development of college and career plans for all 9 grade students; 

Attend the AP Achievement Institute (APAT) in June 2012; 

Assist school leaders to implement SAT/PSAT testing in all high schools; 

Support recruitment of students for SAT testing; and 


mo ao 


Expand AP opportunities, particularly for underrepresented students 


In the initial grant proposal, there were to be two CR Specialists in the first year, with one in the 
high schools and one in the middle schools. Subsequently there would be a gradual ramp-up to six 
CR Specialists, with 3 FTEs in the high schools and 2.5 FTEs in the middle schools by the third 
year. The initial 2012-13 Project Management Plan Summary, however, recommended that OUSD 
establish 5.5 FTEs in the first year. Two FTEs were to staff the seven middle schools, and 3.5 FTEs 
would staff the eight comprehensive high schools. This plan could only be enacted if the grant funds 
originally budgeted across four years were spent in two years rather than four. At this burn rate, the 
13 grant budget would only cover salaries of the 5.5 CR Specialists through the end of the 2013-14 
school year. Additional money would need to be secured in order to sustain the program at this 
level. (It is not clear why the 2012-13 Management Plan differed from what was originally proposed 
in the grant regarding the roll-out of the CR Specialists.) 


b. Create a rigorous curriculum and instruction for all students — specifically linked to provision of 


professional development to teachers, principals, counselors, and administrators 


In June 2012, College Board invited 30 OUSD English and History AP teachers and eight OUSD 
high school principals to an intensive four-day Advanced Placement Achievement Institute (APAT) 
training. However, only nine attended this APAI training. According to key stakeholder interviews, 
there were three possible reasons for the low attendance: 1) other competing professional 
development sessions were scheduled for the same week; 2) the district did not officially prioritize 
APAI over other options available to teachers and principals; and 3) College Board and high-level 
district staff did not communicate effectively with operational staff and departments that were most 
likely to push for teacher and principal attendance (i.e., OUSD staff who were leading college 
readiness and/or AP activities at the high school level). 

The 2012-13 Management Plan did not specifically address a commonly taught district curriculum. 
(It was, however, spoken to in a later version of the management plan, which was not approved by 
the Department of Education until November 2012.) 


c. Develop a clear accountability system linked to the evaluation and 13 research study and to support 
each high school in achieving its college and career readiness goals by establishing and tracking 
performance indicators 


The final goal of clear accountability was to support a district-wide culture of data-driven decision- 
making. This use of data would help the project team track their progress during the Oakland 
Accelerates implementation. The College Board’s Sr. Director, Research & Analytics began 
discussions and forming relationships in order to support OUSD to use the performance measures 
defined in the grant application to track progress towards goals and outcomes. 
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4. Oakland Accelerates/EXCELerator Partnership 

According to program documents, College Board had placed an on-site team in Oakland in February 
2012 (N.B., this appears to have primarily been B.K. who was provided office space next to the 
High School Executive Officer who was closely involved in the creation of the 13 grant.) By May 29, 
2012, B.K. created an initial 13 Oakland Accelerates/ EXCELerator Partnership summaty which 
detailed the responsibilities of both OUSD and College Board teams. It took until November 16, 
2012 for OUSD and College Board to finalize and agree to this Partnership document. 


During the January-June start-up phase, a number of OUSD and College Board personnel made up 
the initial partnership team. However, it took OUSD until June 2012, to assign C.P. as the Oakland 
Accelerates Project Director, officially starting August 1, 2012. College Board was unable to hire the 
full-time EXCELerator Implementation Manager until December 2012. Only one of the College 
Board personnel lived in the Bay Area, meaning that the initial College Board partnership team was 
either calling or flying in for meetings. This may have had an impact on the speed of the start-up 
phase, since critical members of the partnership were often not available in-person to build 
momentum on the project. This created some barriers to communication and undermined the initial 
building of trust and collaboration with the OUSD team. 


Based on key informant feedback, defining the Oakland Accelerates/EXCELerator partnership and 
clarifying strategies and actions in the 2012-13 Project Plan was “the greatest challenge” and 
“required copious amounts of meeting time.” Meeting minutes made it clear that this was a group of 
very committed and well-informed individuals with very different and sometimes philosophically 
opposing views on what it meant for a student to be “college ready.” For example, sometimes 
lengthy discussions would took place on whether or not encouraging more students (some of whom 
might be very underprepared) to take PSAT, SAT, and AP tests and AP coursework was a worthy 
goal. Some felt that these outcomes were evidence that OUSD was effectively supporting students 
to be more college ready. Whereas others believed that such outcomes could have a personal and 
devastating cost if individual students were pushed to take such tests or courses before they were 
ready, and this approach may ultimately discourage students from pursuing a college education. 
These were not simple issues to resolve, and in due course, decisions were made to take actions on 
which consensus was not reached. 


5. EXCELerator Project Manager 

The full-time EXCELerator Implementation Manager was not hired by College Board until 
December 2012. The grant application called for the EXCELerator Project Manager to provide 
leadership for implementation of EXCELerator activities and for delivering professional 
development to all audiences. Therefore, several strategies were delayed until that position was filled. 
However during this period, College Board finalized a job description for the position. 


6-7. CR Specialist Team 

Although the grant application specified that the CR Specialists would be assigned immediately and 
would work with the start-up team, the reality was that OUSD was unable to assign/hire individuals 
as CR Specialists so quickly. A first barrier was that the 13 grant began in January 2012 — in the 
middle of the school year and past the point at which staff could be easily re-assigned or hired. 
Moreover, the Cooperative Agreement between OUSD Board of Education and the Department of 
Education was not finalized until the end of March 2012. Therefore there was no budget available to 
pay staff. The members of the partnership also shared differences of opinion about the specific 
qualifications and roles of the CR Specialist position. As a result, OUSD planned to assign the CR 
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Specialists later starting in the 2012-13 school year, and all of the start-up activities (e.¢., staff 
training) were moved to summer and the start of the next school year. 


8. Evaluation Activities 
During this January to June start-up phase, HTA began working on the grant-mandated evaluation 
plan. (An independent evaluation with cooperation of the evaluator and the TA contractor was a 
requirement of the cooperative agreement signed by OUSD on February 16, 2012.) The National 
Evaluator (Abt Associates) had not yet been contracted to conduct the national evaluation or to 
provide technical assistance to evaluators. Therefore, HTA developed an initial evaluation plan 
without the benefit of technical assistance or of any specific guidelines related to the evaluation plan. 
Senior Associate, Dr. M.B of HTA was assigned the role of lead evaluator. Associate Ruthie Chang 
was assigned to support the evaluation as well. 


Implementation Phase®: July 2012-June 2014 


Table 2. Year 1 (SY 2012-13) Partnership Team Members 


Name Title/Position Organization Funding Source 
A.M. Network Exec. Officer, High Schools OUSD OUSD 

CLP. Project Director, Oakland Accelerates OUSD 13 Grant 

B,J. CR Spec. 1 OUSD 13 Grant 

L.H. CR Spec. 2 OUSD 13 Grant 

T.S. CR Spec. 3 OUSD 13 Grant 

R.P. CR Spec. 4 (PT) OUSD 13 Grant 

T.P. CR Spec. 5 (contracted as needed) OUSD 13 Grant 

Unk. Math Spec. CR Spec. 6 (contracted as needed) OUSD 13 Grant 

G.L. Director, College/Career Readiness OUSD OUSD, Irvine Grant 
*M.H. Implementation Manager, EXCELerator College Board 13 Grant 

DW. St. Director, Research & Analytics College Board College Board 
*BLK. St. Director, College Readiness Initiav. College Board College Board 
L.G. Exec. Director, Implementation College Board College Board 
M.B. Evaluator HTA 13 Grant 

Ruthie Chang Evaluator HTA 13 Grant 


*Names in bold indicate a new team member 


Proposed Program Activities 
1. Establish an effective Oakland Accelerates/EXCELerator Partnership 


Implement strategies identified in the Strategic Work Plan 


Launch the College Readiness Specialist team throughout the district 


2 
3 
4. Provide implementation support (from College Board) 
iS 


Reseatch/Evaluation: Data Collection 


® As described in the grant application, the first two full school years of implementation were to be the “high touch 
phase” where the bulk of activities and resources would be devoted. 
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Program Year 1 (2012-13): Adherence to Proposed Program Activities 


1. Oakland Accelerates/EXCELerator Partnership 

In the 2012-13 school year, the Oakland Accelerates/ EXCELerator Partnership team was made up 
of fifteen members (see Table 2) — although B.K. left in February 2013, and M.H. was brought on in 
December 2012. B.K. was replaced by L.G. and D.W., who had been involved in the start-up phase, 
joined the team when his expertise was requested — especially in regards to improving data systems 
and working closely with R.P. who was assigned to be the Data CR Specialist. Aside from M.H., 
none of the College Board personnel lived in the Bay Area, and they had to either call in or fly in for 
the meetings. When M.H. moved into office space at McClymonds High School, College Board 
became mote effective in moving the project forward during this first school year. However, she had 
limited support and direction as the main EXCELerator leads were located outside of California, 
and she needed to call or email them for guidance. 

While a 2012-13 work plan was developed in June 2012, the Partnership continued to revise and 
further operationalize the work plan to provide a greater level of detail and specific action items. The 
2012-13 work plan underwent a number of revisions and was not approved by the Department of 
Education until November 2012. Due to these delays, the Fall of 2012 became a de facto start-up 
phase, as referenced by C.P. in the APR submitted in April 2013. 


C.P. was assigned as the Oakland Accelerates Project Director following the start-up phase, officially 
beginning her role as the Project Director starting on July 1, 2012. B.K. oversaw the hiring of a full- 
time Implementation Manager, M.H., who was to be the College Board’s “EXCELerator Project 
Manager.” She was to be responsible for coordinating College Board workshops and trainings, 
facilitating meetings with OUSD leadership, and providing insight on the roles and responsibilities 
of the CR Specialists. Bringing her expertise in school reform, Ms. Hudson also received brief 
training on the EXCELerator model from College Board. In the position, College Board did not 
emphasize a need for prior experience on the EXCELerator model, with College Board in general, 
nor any prior experience working in OUSD. 


Two full-time CR Specialists (L.H. and B,J.) began to work on August 1, 2012, and two part-time 
CR Specialists (T.S. and R.P.) started on September 1, 2012. Two had prior classroom experience, 
and the other two did not. (However, these two did have experience providing leadership coaching 
on college readiness.) Two additional CR Specialists briefly provided math coaching support at three 
high schools and one middle school as part of the College Board Springboard program. 


2-3. Implementing strategies outlined in the Strategic Work Plan 


a. Build acollege and career readiness district infrastructure 


During this first year of implementation, the Oakland Accelerates/EXCELerator project leaders 
emphasized that the best way to build a college readiness district infrastructure was to “change the 
behaviors and practices of every adult in the system so they are supporting college and career 
readiness of all students.” And the work of the CR Specialist Team was considered a vital strategy in 
accomplishing this ambitious mission. This was a change of direction from the District Diagnostic 
which called for the creation of goals, policies, processes, and practices — system-level infrastructure 
—as the means to create a culture of college readiness. 
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One of the high school CR Specialists served primarily as college access and readiness support to 
individual high schools. The other high school CR Specialist’s role aligned more closely with the 
grant description which called for a coach supporting teachers, counselors, and school and district 
administrators in implementing instructional and student support practices and programs necessary 
to help students graduate ready for post-secondary success. Both Specialists also reported facilitating 
professional development and working with administrators to improve monitoring practices and to 
identify needed student supports. One of the middle school CR Specialists was dedicated specifically 
to supporting the implementation of College Board’s Springboard ELA curriculum. Another CR 
Specialist was tasked with all of the required data analysis, rather than having each of the CR 
Specialists engage in this task, as originally proposed. In general in the first school year of 
implementation, each CR Specialist was given great freedom to develop their workload and perform 
their role as they saw fit. As a result, the activities conducted by each CR Specialist varied greatly 
with little collaboration or single direction. 


According to the 2012 Annual Performance Report (covering January- December), there was an 
internal debate about the specific roles and responsibilities of the CR Specialists. For example, the 
EXCELerator staff believed that educators would be the best suited to be CR specialists, whereas 
the Oakland Accelerates team wanted to consider individuals with college readiness and/or 
leadership coaching background who did not necessarily have classroom experience. These 
differences in opinion had begun in the start-up phase and continued into this first year of 
implementation. 


The Partnership team ultimately decided to divide the roles of the CR Specialists by their skill set, 
rather than simply by their middle or high school orientation. These skill sets generally aligned with 
the District Diagnostic’s recommendations as it was not possible to find individuals skilled in all 
three areas (see below) as designed in the grant. In addition, the strategic work plan developed by 
College Board and OUSD called for leveraging the work already occurring in the middle schools 
ai the SpringBoard program. 

. Two CR Specialists -- College Readiness and Access Services: Skills in developing programs/services, 
collaborating with community and college access partners, and supporting schools to provide school- 
wide college readiness programs for all students 

2. Two CR Specialists -- Curriculum, Assessments, & Instruction: Skills in developing and mapping 
curriculum, creating assessments, and coaching teachers on instruction 

3. One CR Specialist -- Data Support: Skills in managing databases, analyzing data for meaning, and 
using data to manage projects 


According to the grant application, all of the CR Specialists were to be trained in the EXCELerator 
change framework, but due to delayed hiring of the EXCELerator Implementation Manager, this 
training did not occur until March 2013 as part of a 2-day “College Readiness 101” workshop. There 
were no formal “EXCELerator” change framework workshops or professional development 
provided as per the grant application. The Oakland Accelerates Project Manager reported that if the 
training had occurred earlier in the year, it would have helped to frame the CR Specialist’s work for 
the school year. 
Despite the aforementioned challenges, the following accomplishments occurted as part of this 
infrastructure strategy in 2012-13: 

e = =February 2013 — College Board hosted an Oakland Accelerates Forum which included a presentation 

on College Readiness 101, a luncheon, and data presentation. It was attended by over 80 Oakland 


Prepared by Hatchuel Tabernik and Associates Page | 13 


community members, business partners, city government officials (including the Mayor), OUSD and 
College Board staff. This was considered by the Partnership to be the official “launch” of the 
Oakland Accelerates work. 


e@ March 2013 — College Readiness 101 professional development provided to CR Specialists. 
e =April 2013 — CR Specialists (with College Board’s help) piloted a 4-year student College and Career 
Plan (CCP) template for 9" grade students at three high schools. 


In June 2013, A.M., the High School Executive Officer, with consultation from the Oakland 
Accelerates staff, developed Advanced Placement Policies. These policies were based on College 
Board’s AP Equity and Access Statement’ and were designed to be part of a larger college and career 
readiness package that would provide for open access, remove participation barriers, and ensure all 
AP students take the AP exam. In addition to these policies, the package included draft 
College/Career Readiness Goals for Grades 6-12 and a draft Graduate Profile. Plans were in place 
to submit new Advanced Placement Policies to the Board of Education for approval before June 
2013; however, with A.M.’s departure and other leadership changes, this did not happen, and the 
policies continued to be revised and modified in the 2013-14 school year. 


As mentioned earlier, building a common understanding and shared vision of college and career 
readiness was one of the goals in the start-up phase about which there was much debate. According 
to the 2012-13 plan, which was updated and sent to the Department of Education in June 2013, the 
Partnership had by then agreed on a definition of college readiness to mean "ready to succeed in 
post-secondary education without the need for remediation." The Partnership also committed: “For 
this goal to become a core value in the culture of our district and every school, [it will require] the 
continuing work of the Oakland Accelerates team.” 


b. Create a rigorous curriculum and instruction for all students 


Prom July 1, 2012 through June 30, 2013, the Partnership provided a number of professional 
development opportunities for OUSD personnel. Based on the evaluator’s review of the records 
from the first school year, the following professional development occurred. 

e October 2012 — OUSD created a cohort of middle and high schools to implement the Springboard 
Pre-AP curriculum and partnered with College Board to provide professional development for 
principals, teachers and teacher leaders at these schools and for district Springboard instructional 
coaches. 

e November 2012 — To address the learning needs of OUSD’s diverse students, in particular students 
who are not meeting grade level standards in core academic areas, Educators for Social Responsibility 
(ESR) was contracted to train eight OUSD high school principals in a series of professional 
development sessions covering Response to Intervention and Instruction. 

e@ =March 2013 — A two-day condensed version of the four-day APAI summer training called QuickStart 
was offered to the AP teachers and principals who did not attend during the summer. In the end, a 
total of 23 AP teachers and principals were trained — either during the summer or during the 
QuickStart session. 


7 http://professionals.collegeboard.com/k-12/assessment/ap/equity 
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During this school year, OUSD was also working to build awareness regarding Common Core and 
the shifts in practice that would be necessary for implementing Common Core by providing 
professional development district-wide. To a large degree, this work was leveraged by the 
Partnership to address Oakland Accelerates’ goals of vertically aligning curriculum based on 
Common Core Standards which ate more oriented to college and career readiness. Some College 
Board professional development opportunities, originally planned for the summer of 2013, were 
delayed to the following school year because OUSD had already prioritized a number of Common 
Core-related professional development, and the Partnership did not want to repeat what had 
happened in the previous summer with very low attendance at the 2012 APAI. 


Finally, the 2012-13 work plan had been expanded to include the goal of developing more pathways 
for AP teachers and aspiring AP teachers by expanding the APAI training to non-AP teachers, and 
AP teachers in math and science. Unfortunately, the Partnership was unable to gain much traction 
on this goal due to low attendance at the Summer 2012 APAI and the delay of the Summer 2013 
APAIT due to conflicting Common Cote trainings. 


c. Develop a clear accountability system 


Efforts towards an accountability system this year focused on giving teachers and administrators 
tools to prepare for tests and interpret student PSAT, SAT, and AP data. From HTA’s review of the 
records from the 2012-13 school year, the following accomplishments also accrued in association 
with this infrastructure strategy: 

e October 2012 — OUSD paid for and administered the PSAT to 1,487 10" graders in eight high 
schools (70% of all 10" graders in those schools) 

e September 2012 — College Board provided an additional 300 SAT test fee waivers (in addition to the 
waivers that they usually provide) to encourage low-income youth to take the test. 

e Spring 2013 — CR Specialists, in coordination with school administrators and College Board experts, 
provided follow-up APAI professional development to support teachers to prepare students for 
upcoming AP exams. 

oO Redelivery (March 2013) 
oO Follow-up (April 2013) 
o Classroom obsetvations (March-April 2013) 

e = Spring 2013 — College Board worked closely with OUSD’s Research, Assessment, and Data (RAD) 
division and the CR Specialists to improve the district data systems, especially as it related to cleaning 
and merging AP, SAT, and PSAT test data with OUSD student data. 


To model how administrators should use PSAT results to expand AP course offerings in individual 
schools and to recruit students into AP courses, the Implementation Manager engaged in several 
data debriefings for district administrators and high school principals: 

e =March 2013 — District level debriefing of PSAT Test Results 


e = =April 2013 — Principal debriefing of PSAT Test Results 


4. Implementation Support via Professional Development 

During the 2012-13 school year, College Board also conducted a number of principal, counselor, 
and administrator professional development trainings and workshops. Interestingly, many of these 
workshops were not called for in the Partnership project plan. However, they were listed as 
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“implementation support” to be provided as part of College Board’s Partnership Agreement with 
OUSD. 

e September 2012 — PSAT® training for test coordinators 

e September 2012 — AP Potential and SOAS? training for principals 

e October 2012 — Understanding PSAT training for testing coordinators and CR Specialists 

e November 2012 — Fall Counselor workshop 

e December 2012 — Fall Counselor/College and Career Readiness workshop for counselors, college 

readiness specialists, and community-based organizations 

e January 2013 — PSAT Student Tools training for principals and testing coordinators 

e January 2013 — PSAT/NMSQT — My College QuickStart training 

e =6February 2013 — Summary of Answers and Skills training for principals 

e = =Match 2013 — SAT Tools Training 

e = =April 2013 — AP Potential training for Oakland Accelerates team 

e =April 2013 — APAI Prepare Your Students for the AP Exam workshop 


5. Evaluation Activities 

HTA attended Partnership meetings, and outlined the data collection methods and measurements, 
including detailing NEi3’s “contrast tools” or outcome measurement parameters, and identified an 
appropriate comparison district as requited by the NEi3 team and the Department of Education for 
the “Impact Study.” HTA collaborated closely with Partnership members to define the indicators of 
fidelity for the “Implementation Study” element of the Research and Evaluation Design. The 
Research and Evaluation Design document was submitted for two rounds of review and approval in 
October 2012 and in May 2013. An HTA Senior Associate attended the 13 Project Directors’ 
Conference in Washington, DC in May 2013 along with the Project Director and two CR Specialists. 
The HTA Director of Research and Evaluation took over the lead evaluator role in May when the 
Senior Associate left the firm. 

A component product of the Research and Evaluation Design was an Oakland Accelerates Logic 
Model which articulated how the intervention was meant to work. (See below for logic model, 
Figure 2.) 


8 The PSAT is also known as the NMSQT (National Merit Scholarship Qualifying Test) 
° The SOAS is the Summary of Answers and Skills test 
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District Diagnostic: OUSD performs 
a district diagnostic to inform the 
work plan — as seen below in #1-3 


1) Professional Development for 
quality instruction, rigorous 
coursework, and capacity building 


for expanding college readiness: 
CB provides PD through 


workshops, webinars, 
conferences, summer institutes, 
and trainings; and follow-up for 
OUSD leaders, counselors, 
teachers, and staff. 


2) CRS Coaching Support: 

CB provides CRS with coaching, 
mentoring, technical assistance, 
training to support OUSD staff, 
data access, educator knowledge, 
and college readiness programs. 
CB provides CRS with resources to 
support OUSD college going 
culture. 


3) Capacity and Policy Guidance: 
CB provides OUSD with technical 


assistance and recommendations to 
develop structures, capacities, 
policy, and culture to support 
college going culture. 


Figure 2: Oakland Accelerates Logic Model 


# of trainings provided to 
teachers, counselors, 
leaders and staff ; 

# of staff receiving follow up 
services. 


# of formal coaching, 
mentoring sessions, and 
trainings CRS receive; 

# of CRS meetings held; 
# of times resources 
accessed by CRS. 


# of meetings held with 
OUSD leadership; 

# of trainings to support 
curriculum alignment. 


# of resources provided to 
OUSD for parents, staff, 
teachers, students, leaders; 
# of information sessions 
held for parents. 


Ee) Mel-le-Mocy-Toal la) 


monitoring, and 
implementation meetings. 


Staff, teachers, counselors, and 
leaders are skilled in providing a 
rigorous education and creating a 
college going culture. 


CR Spec. provide effective coaching 
to staff and identify and address 
college readiness issues. 


OUSD leadership, teachers and staff 
are knowledgeable about college 
going policy. 

OUSD leadership, teachers and staff 
have the capacity to support 
college going culture. 

OUSD aligns curriculum. 


Parents, students, and teachers will 
become more knowledgeable about 
the processes, requirements, and 
resources available for getting into 


| college. 


OUSD leaders and CB will have 
understanding of how they are 
adhering to program model, what 
their strengths and weaknesses are 
and feel more accountable. 


AP teachers will have the skills and 
capacity to enable underrepresented 
students to succeed in AP classes. 


Teacher attitudes, beliefs, and 
perceptions towards under- 
represented students’ college 
attainment will improve. 
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Program Year 2 (2013-14:) Adherence to Proposed Program Activities 


Table 3. Year 2 (SY 2013-14) Partnership Team Members 


Name Title/Position Organization Funding Source 

G.L. Project Director, Oakland Accelerates and | OUSD OUSD, i3 Grant 
Director, College/ Career Readiness 

M.H. Implementation Manager, EXCELerator College Board 13 Grant 

D.K. CR Spec. 1 OUSD 13 Grant 

L.H. CR Spec. 2 OUSD 13 Grant 

KS. CR Spec. 3 OUSD 13 Grant, OUSD 

T.R. Content CR Specialist 1 OUSD 13 Grant, OUSD 

K.B. Content CR Specialist 2 OUSD 13 Grant, OUSD 

C.L. Content CR Specialist 3 OUSD 13 Grant, OUSD 

LJ. Leadership Manager and Principal Coach OUSD 13 Grant, OUSD 

DW. St. Director, Research & Analytics College Board College Board 
(contract) 

L.D. Sr. District Director, Western Region College Board College Board 

Danielle Toussaint Evaluator HTA 13 Grant 

Ruthie Chang Evaluator HTA 13 Grant 


*Names in bold indicate a new team member 


1. Oakland Accelerates/EXCELerator Partnership 

In the summer of 2013, OUSD officially changed Project Directors from C.P. to G.L., Director of 
College and Career, under the supervision of the Leadership, Curriculum, and Instruction (LCI) 
office in order to leverage existing administrative and financial resources. The CR Specialist Team 
also came under the purview of the LCI office, and attended LCI department meetings in addition 
to Oakland Accelerates meetings. M.H. continued as the College Board Implementation Manager, 
collaborating with the Project Director and facilitating meetings with the entire implementation 
team. These regular meetings included all CR and Content Specialists as well as the Project Director. 
Of the six CR Specialists, only one remained for the second program year. (The others had taken 
other positions within or outside of the district.) One of the two new CR Specialist was hired to 
have a high school presence in supporting college access and readiness and the another to 
coordinate the data management goals of the project. In addition in this year (and as decided in the 
prior school year), three CR Content Specialists were hired to cover Math, History and English and 
provide more specific instructional support for teachers, observing and modeling instruction. 
Although there were fewer College Board staff on the Partnership than in the prior year, the 
Implementation Manager was on site at OUSD full time, helping to build mote positive relations 
between the College Board and OUSD. Reviewing program documents and meeting minutes, HTA 
noted that there was less tension about decision-making, and the partnership was able to dive into 
the activities spelled out in the 2013-14 work plan in addition to building on work accomplished in 
2012-13. In addition to the Implementation Manager being on site, she was also the only person 
devoted to the Oakland Accelerates project full time. She leveraged and guided stakeholders who 
were often working on other projects and commitments. 


2-3. Implementing Strategies in the Strategic Work Plan 


a. Build acollege and career readiness district infrastructure 
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The work of the CR Specialist Team continued to be the primary activity for this strategy. The three 
new CR Specialists were recruited specifically to be “content” specialists, as it was decided to meet 
instructional support needs during the project meetings. 

1. ‘Two CR Specialists — College Readiness and Access Services: Skills in developing programs/services, 
collaborating with community and college access partners, and supporting schools to provide school- 
wide college readiness programs for all students 

2. Three CR Specialists (Content Specialists) - Content Specialists in curriculum, assessment and 
instruction; skills in developing and mapping curriculum, creating assessments, and coaching teachers 
on instruction in the areas of Math, History, and English 

3. One CR Specialist — Data Support: Skills in managing databases, analyzing data for meaning, and 
managing projects utilizing data 


The meeting minutes and stakeholder interviews indicate that much more guidance was provided in 
2013-14 to CR Specialists as to their day-to-day tasks and responsibilities. Highlights of CR 
Specialists’ work this year included rolling out the College and Career Readiness Plans (CCPs) on a 
digital platform for all 9" graders. The CCP was designed: 1) to track individual student progress on 
the “a-g” requirements for UC/CSU admission and OUSD graduation requirements; 2) to provide 
information about how students can prepare themselves for success throughout high school; and 3) 
to help students access all available resources, connecting with site staff and partners, and learning 
more deeply about what high school will entail for them as they step along the path to college. In 
November 2013, CR Specialists developed a digital platform for the 9° grade CCPs in partnership 
with ConnectEd (a non-profit organization which supports high school reform efforts centered on 
career-themed academies and pathways) and used it at the three high schools pilot sites beginning in 
the 2012-13 school year. By February 2014, the CR Specialists had rolled out the 9" grade CCP 
digital platform at the remaining five high schools and led presentations to students on how to 
create their own CCPs. 


Figure 3: OUSD 9* grade students creating college & career plans, 2011-14 


Zero 


2011-12 2012-13 2013-14 
Data Source: Oakland Accelerates Program records, 2012-13 & 2013-14 


In Fall 2013, CR Specialists leveraged the services of college access providers assigned to their sites 
(c.g. East Bay Consortium, East Bay College Fund, Mills Educational Talent Search, Upward Bound, 
Destination College Advising Corps) to support SAT preparation and registration, college visits, 
college applications and personal statements, financial aid information and resources, and other key 
college readiness activities. At the three largest high school sites, this coordination came in the form 
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of formal college collaboratives that meet regularly to plan activities, share resources, and review the 
efficacy of the team’s efforts. CR Specialists also assisted in the district’s communication plan 
regarding PSAT administration to all 10" graders in October 2013, as described in more detail 
below. 


Highlights of Content Specialists’ work included coaching both AP and pre-AP teachers, facilitating 
AP teacher leader meetings, and vertical articulation of pre-AP content leading to AP opportunities 
for students. The Science CR Specialist in particular focused on curricular and coaching strategy 
development by adopting a district-wide science sequence for high school, to ensure equitable access 
for all high school students to AP-level science courses by 12" grade (if not sooner). This sequence 
adoption was supported significantly by the adoption of an OUSD 9" grade biology curriculum 
which supports standardized implementation and support across all school sites. 


Also in the fall of 2013, OUSD developed a clear policy that all 10° graders, with the exception of 
the severely cognitively disabled, would participate in PSAT testing. In addition, the district created a 
Public Service Announcement and broadcast video to promote and encourage 10" grade 
participation and inform the Oakland residents of the district’s efforts. 


Work on developing Advanced Placement Policies for the district continued in the 2013-14 school 
year. As described earlier, the out-going High School Network Executive Officer had planned to 
submit the package to the Board in June 2013. However, this did not happen — most likely due to 
the changeover in administration with the departure of Superintendent T.S. According to College 
Board’s final report (June 2014), the Oakland Accelerates team continued to work on the package 
incorporating feedback from the new administration, and College Board provided OUSD with data 
analysis and information to provide a rationale for the adoption of the revised AP policy. The policy 
was set to be presented to the board in the fall of 2014. 


Finally, the Partnership continued to build a district infrastructure promoting college readiness by 
focusing on the promotion of AP equity (e.g., access to and success in AP classes by all students, 
especially those from underrepresented groups). Led by the Implementation Manager and the CR 
Specialists, all high schools were provided with their sites’ AP potential data (by subject area) to 
increase AP course enrollment across the board — with a particular focus upon students of color 
who have historically been underrepresented in these courses. AP potential data is dertved from 
PSAT scores which help schools identify additional students who might benefit from AP enrollment 
and who may otherwise not have been identified. The Partnership worked to ensure that principals 
would have the AP potential data early in the master scheduling process to allow for more 
intentional decision-making about course offerings, as well as to make formal requests (if necessary) 
for teacher training and syllabus/course development support prior to the next school year. 


b. Create a rigorous curriculum and instruction for all students 


While the implementation of SpringBoard in the middle schools was one of the Partnership’s areas 
of focus in the 2012-13 school year, the focus shifted to general pre-AP coursework in the high 
schools during the 2013-14 school year. Much of this work involved content support and coaching 
provided by the three CR Content Specialists who were able to help both pre-AP and AP 
classrooms in the eight high schools. College Board also provided consultation to pre-AP teachers in 
vertical articulation of strategies meant to prepare students in non-AP science and world history 


Prepared by Hatchuel Tabernik and Associates Page | 20 


classes for the rigors of AP Science and AP World History. College Board also provided a series of 
AP workshops for AP teachers in different content areas as outlined below. 


Yet according to College Board’s final report (June 2014), attendance at the College Board AP 
trainings remained low; a total of 34 of 68 eligible AP teachers attended at least one AP training in 
the 2013-14 school year. The Implementation Manager speculated that the reason for the low 
attendance by current AP teachers was two-fold: 1) the leadership structure for communication with 
high school principals (many of them new) was not well-established; and 2) experienced “veteran” 
AP teachers chose not to attend the training, perhaps believing that they already had the experience 
needed to teach an AP class. As a result, a robust professional learning community among many AP 
teachers could not be well established. 


Based on review of the 2013-14 records, the infrastructure strategy yielded the following outputs 
supporting curriculum and instruction in 2013-14: 
e = October 2013 — AP Syllabus Writing Workshop 
e November 2013 — Series of eight workshops targeting AP teachers in various content areas (Le., 
Environmental Science, Calculus, English Literature, US History, English Language, World History, 
Biology, and World Language) 
e = Multiple dates — Follow-up Coaching (ELA, Math, and Science) for select AP teachers who 
participated in November 2013 workshops 


c. Develop a clear accountability system 


Figure 4: Three-year growth in 10" grade students taking PSAT, 2011-14 
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Data Source: OUSD student academic data, 2011-12 & 2013-14 


Based on 2013-14 school year records, the following activities occurred to advance the 
accountability strategy: 
e September/October 2013 — A districtwide communications plan was developed to inform all 10% 
grade families of the PSAT administration to be held in the Fall. The district sent letters (translated 
into five languages) and conducted robocalls to phone numbers listed for each family. In addition, a 
PSA was created to promote OUSD’s new policy on PSAT administration. 
e September/October 2013 — CR Specialists and College Board provided support to site PSAT 
coordinators and proctors, and assisted sites with PSAT exam administration planning. 
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e October 2013 — OUSD paid for and administered the PSAT to 10% graders in all eight high schools. 
e The CR Data Specialist worked closely with College Board to clean and match College Board data 
(e.g., PSAT and SAT scores) with OUSD student data to support on-going program improvement. 


In order to sustain the work, the Implementation Manager and the CR Specialists conducted 
extensive walk-throughs and surveys at the eight high schools in October 2013. They used a detailed 
rubric to determine the greatest strengths and areas of continued need regarding college readiness. 
Using this information, the Partnership developed a sustainability plan for calendar year 2014, which 
they have continued to revise and update in the 2014-15 school year. 


4. Implementation Support via Professional Development 
As in the prior year, College Board also conducted a number of principal, counselor, and 
administrator professional development trainings and workshops to help support the 
implementation. 

e July 2013 — AP Equity and Annual Conference (three days) in Las Vegas 

e September 2013 — PSAT coordinators’ training workshop 

e September 2013 — PSAT proctor training 

e September 2013 — Fall counselor workshop 

e September 2013 — AP coaching and professional learning planning followed by two days of AP 

classroom walkthroughs and feedback via a structured observation rubric 

e October 2013 — Counselor data-based professional learning training 

e January/February 2014 — PSAT data and tools workshop 

e February 2014 — Counselor professional learning training (follow-up to Oct 713 training) 

e = March 2014 — Preparing students for the AP Exam (targeting AP teachers) 


5. Evaluation Activities 

The HTA evaluator continued to attend regular meetings with the team and work closely with the 
Implementation Manager to collect fidelity of implementation data on a quarterly basis. HTA 
tracked fidelity of implementation during the two program years using the fidelity matrix submitted 
to and approved by NEi3 (see Appendix). In order to collect data on program implementation, 
HTA developed an easy-to-use tracking spreadsheet of activities, meetings, and those in attendance 
for the Implementation Manager to complete and share with HTA. The Implementation Manager 
also used this document to report out College Board’s implementation progress to the OUSD team. 
In 2013-14, the evaluator supported the Project Director with the analysis and reporting required for 
the annual performance report submitted to the Department of Education in April 2014. Dr. 
Toussaint and Ms. Chang attended the 13 Project Directors’ Conference in Washington, DC in June 
2014 along with the Project Director and two CR Specialists. For the federal research report, the 
NEi3 Analysis and Reporting (AR) Team reviewed the 13 evaluation plan and confirmed the plan 
met the GPRA criteria for #2 and #3. The NEi3 AR Team also registered the final evaluation plan 
as the official plan on record with the NEi3. 
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Monitoring Phase: July 2014 - December 2015 


Proposed Program Activities 
1. Partnership works to secure additional funding to sustain program as well as making and 
implementing any other called-for changes 
2. Monitor student outcomes, adult changes in practice and momentum of program itself 
3. Research/Evaluation: Final Report 


Adherence to Proposed Program Activities 


1. Oakland Accelerates/EXCELerator Partnership & Sustainability 

With the departure of the College Board Implementation Manager, the Partnership between College 
Board and OUSD came to an end. 

Oakland Accelerates continued in 2014-15 with a consistent Project Director and three CR 
Specialists maintaining their roles. Without a full-time Implementation Manager to coordinate and 
hold project goals at the forefront and with a Project Director who is continuing a full-time position 
as the Director of Linked Learning, meetings of the diminished Oakland Accelerates team were less 
regular and the CR Specialists were more autonomous in the services they provided to each of the 
school sites. G.L. continued to align and leverage Oakland Accelerates with complementary college 
and career readiness initiatives within her Linked Learning Office and within other OUSD offices. 
In August 2014, the Oakland Accelerates team met with the Department of Education’s grantee 
implementation technical assistance provider (V.S. from Westat) to document, codify and align all 
Oakland Accelerates activities with OUSD’s Graduate Profile. It is intended that this document will 
be used by high school sites to plan college readiness activities throughout the school year. 


After an extensive national search, the OUSD Board of Education unanimously selected A.W. as the 
new superintendent effective July 1, 2014. By November 2014, he produced a 5-year strategic plan 
for the district, including his vision of college readiness and preparation needed for OUSD 
students.'” In addition, he re-organized many of the OUSD departments under newly created 
administrative departments, and hired new executives to head these departments. 


In May 2015, the new Deputy Chief of Post-Secondary Readiness announced that A.A., the new 
Manager of College Readiness, would take over as Oakland Accelerates Project Director starting in 
July 2015, and the CR Specialists who were to be funded in 2015-16 (due to carryover of unspent 
grant funds) would report to her for all future Oakland Accelerates activities. At the time of this 
report, she has stated being committed to continuing the work already achieved. However, it is not 
yet clear whether she will build on the plan originally developed by College Board and OUSD in 
2012 (and subsequently revised) as the basis for the Oakland Accelerates program moving forward, 
or if the new administration led by Superintendent A.W. will want to build off of these efforts and 
craft a new college readiness program based on their own perspectives and priorities. 

In interviews with Oakland Accelerates team members in May 2015, respondents were hopeful that 
the CR Specialist positions would be continued as envisioned during the course of the grant-funded 
program. Several respondents mentioned optimistically the recent voter-approved City of Oakland 
Measure N, a parcel tax which directs OUSD to better fund school programs designed to prepare 


0 Oakland Unified School District. (2014, November). Pathway to Excellence 2015-2020: Every Student Thrives 
(Strategic Plan). Oakland, CA: Author. Retrieved from http://www.ousd.k12.ca.us/Page/5 
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students for college and career. They hoped that this additional money would be used to continue 
the work that Oakland Accelerates had started, especially sustaining the CR Specialists. 


2. Monitoring Program Momentum 

New partnerships, initiatives, and goals have formed to build on the college readiness success of the 
previous years. As the 13 grant moves into the “monitoring phase,” program components are being 
re-examined and built upon. Adding on to the success of the customized student-level CCP reports, 
the CR Specialists are now working with the OUSD Quality Accountability and Assessments 
department to create school and pathway level Balanced Scorecard reports that can be used to assess 
the college and career readiness of students, especially when used in concert with Transcript 
Evaluation Service reports out of UC Berkeley and delivered to all high school sites twice a year. CR 
Specialists provided coaching to principals and lead teachers in how to best utilize this data. 


OUSD’s new administration came forth with a vision for continuing to build the common 
understanding of college readiness of the prior two years. In the new five-year OUSD strategic plan, 
the goal of all OUSD students graduating college, career and community ready was to be achieved 
through the development of high quality linked learning pathways through every high school. 
Moreover it was specified that each high school would ensure that graduates meet the OUSD 
Graduate Profile by implementing the four pillars of linked learning: 1) a challenging academic core 
taught through real world relevance; 2) strong career technical programs of study; 3) a scope and 
sequence of work based learning experiences; and 4) individualized and differentiated student 
supports. While the Graduate Profile has not yet been presented to the Board for approval, key 
stakeholders have reported that anecdotally it has become widely accepted by many OUSD 
personnel as of the 2013-14 school year. 


As described earlier, plans were in place to present the modified Advanced Placement Policies 
package (promoting equity and access) to the Board in the fall of 2014 (and the fall of 2013). Yet 
again this did not happen. According to meeting minutes and the APR, since the new 
Superintendent had re-organized district offices to create a new Office of Post-Secondary Readiness 
headed by Chief of Schools A.S. and Deputy Chief B.M. (both new OUSD positions), the Oakland 
Accelerates team wanted to wait for feedback from this new team so they could “put their mark on 
it.” As of the time of this report, it was anticipated that this Advanced Placement Policy package 
would be shepherded through the process for Board approval by the new Manager of College 
Readiness, A.A., who was also assigned to be the Oakland Accelerates Project Director during the 
2015-16 school year. 


According to current OUSD leadership, Measure N will undoubtedly continue much of the work 
built by the Oakland Accelerates staff. There is a deep commitment by the OUSD Leadership team 
to continue supporting the PSAT (at no cost to students) as well as now the SAT (also at no cost to 
students). There also remains a commitment to the continued training of Advanced Placement 
staff. Although Measure N is allocated directly to sites, there is a portion that allows the district to 
use these funds for central-supported efforts, such as the PSAT and AP training, and that is part of 
the Measure N implementation plan. 


3. Evaluation Activities 

HTA continued to attend meetings, and to support OUSD in the APR reporting and analysis 
process. The HTA staff also began to collect the final data (student achievement data, 
teacher/administrator surveys and stakeholder interviews) needed to assess program impact for the 
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final impact evaluation report. The final student outcome data and contrasts are expected to be 
reported to the National Evaluation 13 team by April 2016. Dr. Toussaint and Ms. Chang of HTA 
again attended the 13 Project Directors’ Conference in Washington, DC in June 2015 along with the 
outgoing and incoming Project Directors. 


Conclusion 


Challenges in Start-Up Phase (Jan — June 2012) 

A number of contractual delays created implementation challenges that affected the project timeline. 
The signing of the 13 cooperative agreement was delayed by three months, slowing the beginning of 
the collaboration to build relationships, define roles and responsibilities, and prioritize activities. In 
addition, the Management Plan was finalized nine months late, delaying the execution of the 
strategies called for in the Strategic Work Plan. Further complicating matters, The College Board’s 
contract with OUSD was not finalized until a year and half after the grant was funded, contributing 
to implementation setbacks related to hiring, planning, and initiating programmatic activities. 


These delays disconnected the progress that The College Board made with their District Diagnostic 
needs assessment prior to the start-up phase from the subsequent development of the Oakland 
Accelerates strategic work plan. The implementation team from both OUSD and The College Board 
were expected to take the lead in implementing these work plan activities. Since understanding and 
buy-in of this strategic work plan was not achieved during this start-up phase, the Partnership spent 
much time in the 2012-13 school year revisiting these concepts, adding new information, and taking 
new directions in their planning. 


In addition to delays, program details describing the elements to be implemented were not clearly 
defined. Besides describing how OUSD would create a “college readiness team,” neither the grant 
application nor an early “Road Map” specified exactly how several identified priorities might be 
addressed. For this reason, many Oakland Accelerates activities were developed on an ad hoc basis 
via the collaborative partnership between OUSD implementation team members and The College 
Board’s implementation team between July 2012 through June 2014. 


Challenges in 2012-13 

Difference in opinions developed among members of the partnership around strategies and actions 
proposed in the grant application versus what they personally believed needed to be done to address 
the issues. Perhaps in part because of the lack of agreement and buy-in as well as the delay in 
assigning or hiring key staff, the Oakland Accelerates Project Director considered the Year 1 Work 
Plan produced by College Board to be open to adjustments and modifications, and in the APR, 
referred to the Fall 2012 term to be the actual “start-up phase.” 


One major area of disagreement was whether College Board’s SpringBoard program, which was 
already occurring in the middle schools, should be considered part of the Oakland Accelerates 
program strategy as well. College Board EXCELerator team members, in particular, believed this 
work was vital to the grant’s goals. However, some of the OUSD leadership disagreed — stating that 
the grant was meant to focus on AP courses in high school, not pre-AP courses in middle school. In 
this discussion, even the lead evaluator who was working on developing fidelity measures for the 
project added her perspective that it was clear from the grant application that the project was always 
meant to focus primarily on high school students, and changing the culture of college-going and 
college readiness in that sphere. In the end, there was consensus that including the Springboard 
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program and dedicating CR Specialists to the middle schools did not align with the main goals of 
Oakland Accelerates. 


Another area of discourse centered on the role and responsibilities of the College Readiness (CR) 
Specialists. Some argued that the CR Specialists needed to be focused on rigor as content specialists 
and coaches to AP teachers, whereas others believed the CR Specialists should be focused on 
college preparedness more generally, working with principals and counselors. Some were of the 
opinion that CR Specialists needed to have a classroom teaching background in order to be effective 
in counseling and assisting AP teachers. Others disagreed, believing that experience in leadership 
coaching and mentoring was just as important. There were also logistical requirements that OUSD 
find appropriate CR Specialist staff very quickly. As described earlier, the Partnership team 
ultimately decided to divide the roles of the CR Specialists by individual skill set. OUSD hired from 
within and without the district to form a group with varying backgrounds. 


Another point of discussion focused on the role and responsibilities of the new Implementation 
Manager. As a College Board employee, she was directed to do a set of tasks that the outgoing 
EXCELerator team members as well as the grant had outlined for her. However, being part of a 
partnership with OUSD and working in an unfamiliar school district, she had to be flexible and 
adapt to what OUSD team members wanted. As a result, a number of discussions in early 2013 
centered on what she should or should not be doing as part of the intervention. 


Each of these tension points were directly related to the struggle over the scope and breadth of the 
work to be covered by the Oakland Accelerates program. Everyone on the partnership had a 
different vision on what Oakland Accelerates needed to do in order to achieve its very lofty goals. 
Many members wanted to include and leverage all college readiness initiatives in the district to fall 
under the “umbrella” of the Oakland Accelerates program; however, not all could agree on which 
programs (such as SpringBoard) should actually be included in program activities. According to key 
stakeholder interviews, several members of the Partnership team reported going back over the 
original grant application for guidance on how inclusive the Oakland Accelerates program was 
meant to be. From the evaluator’s perspective, the description of the Oakland Accelerates program 
in the grant application was intentionally vague since the intervention was, by design, dependent on 
the outcomes of the District Diagnostic. But since the District Diagnostic was itself both vague and 
ambitious, many questions on vision and scope remained. 


In essence, the lack of common vision in this Partnership was reflective of the source documents 
from which they were starting and from the executive leaders driving the process. For example, one 
stakeholder involved in the early start-up phase of the project stated their opinion that 
Superintendent T.S. understood the goals of the partnership but those below him did not. As 
Superintendent T.S. became less involved in 2012-13, certainty about program goals became less 
clear. After the departure of A.M. in 2013-14, there was little involvement or input from higher-level 
administration from that point onward. By mid-year 2014-15, there was a common vision from the 
new Superintendent and his newly hired executives — however, it appears that this vision is not 
based on what was established during this program implementation. 


At College Board, meanwhile, a new President and CEO was hired in the fall of 2012, and within a 
few months, the entire EXCELerator project was scrapped. Although College Board assured OUSD 
that they would follow the terms of their contract, the Partnership would no longer have the benefit 
of including experienced EXCELerator staff, who had participated in the District Diagnostic, in 
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future program activities. By February 2013, the five-person College Board team was reduced to a 
new Implementation Manager who was new to College Board as well as to OUSD, as the only full- 
time College Board representative. She received some consulting support from her superior and 
other College Board technical staff as needed. While this was certainly a heavy loss in the first year 
of implementation, key stakeholders reported in later years that the departure of the College Board 
team actually helped the remaining Partnership team (primarily OUSD staff) come to consensus and 
reach agreement on the direction of the Oakland Accelerates program. 


In April 2013 Superintendent T\S., a key partner in the project with a strong relationship with 
College Board, announced his departure effective in June. Following the announcement, OUSD 
Board Member G.Y. was voted to be the acting superintendent for the 2013-14 school year, while 
the Board was conducting a national search for a new superintendent. Later in the school year, the 
High School Network Director, another key partner in the project, announced her retirement 
effective July 2013. Her position was subsequently filled; however, there is no evidence that her 
replacement was actively involved with the Oakland Accelerates team planning or implementation in 
subsequent years. Consequently, the project team lost the members in high-level leadership positions 
at OUSD to maintain the visibility of the work. 


Challenges in 2013-14 

Without the direct line to district leadership as existed in 2012-13 with Superintendent T.S. and the 
High School Network Officer, the project staff struggled with communication and coordination 
between departments and offices. In the opinion of some key stakeholders, many competing 
initiatives operated in parallel with Oakland Accelerates activities, often utilizing Oakland 
Accelerates staff, which made it difficult to establish priority for many program activities called out 
in the Strategic Work plan. 


As of July 2014, College Board ended its participation in the Oakland Accelerates project, arguably 
six months earlier than was originally proposed in the grant. The Implementation Manager was laid 
off by College Board after June 2014, and any College Board support ended along with her 
departure. 


Despite these challenges, the team found greater cohesion in the second year of the project. The 
team, stable since the end of the previous school year, was able to make progress on many of the 
goals of the 2013-14 work plan and function smoothly as a team. The work of the CR Specialists in 
particular were more aligned and coordinated with each other as well as within the greater goals of 
the project than during the prior year. The scope of what the Partnership team was trying to 
accomplish was more focused and attainable than what it had been in the previous year. Because 
many of the key strategic decisions had been decided in the prior year, e.g., roles and responsibilities 
of CR Specialists; defining college readiness district-wide; focusing on only the eight comprehensive 
high schools, rather than including all the middle schools; etc. a more productive 2013-14 
implementation was possible. 


From the evaluation perspective there were several primary factors contributing to this lack of 
common vision: 1) the vague and overly ambitious nature of the Management Plan that resulted 
from the District Diagnostic; 2) the lack of consistent district messaging from executive leadership; 
and 3) extensive turnover in OUSD and College Board staff. 
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Successes 

Despite these challenges, several value-added activities and lessons learned came out of the 
collaborative work on the Oakland Accelerates project. At the beginning of the project, 
communication within OUSD was reported as “siloed” and “byzantine.” Schools operated 
independently with some that had academic counselors and some that did not. Many of those 
interviewed described great difficulty in getting the right people into Partnership meetings or getting 
on the agenda of meetings that those people attend. By 2013-14 however, program staff reported 
being able to collaborate more effectively among different groups of people doing similar work or 
working towards similar goals. Much more cross-team work occurred due in large part to the 
decision to house Oakland Accelerates in the Leadership, Curriculum and Instruction (LCI) office. 
This administrative change meant that efforts could be more efficiently streamlined and 
unduplicated. 


Another area of communication success was in the coordination of the AP Teacher Professional 
Development trainings. During the startup period, only nine of 23 invited teachers attended. By the 
implementation period (2012-14), the challenges around district calendaring were addressed and 
teacher participation increased and feedback was very enthusiastic. While the program team would 
have liked to see the attendance numbers higher, the teachers that did attend reported that they 
found the trainings very beneficial and that they instituted what they learned immediately in their 
classrooms. 


Many interview respondents mentioned that district-wide institutionalization of the PSAT for all 
10°" grade students as one of the greatest Oakland Accelerates successes. The test supports all 
students’ test-taking skills for the SAT and provides rich data to students and teachers to better 
assess strengths and weaknesses to prepare for college and college-ready classes. CR Specialists 
worked with teachers to help them understand and translate PSAT scores into useful knowledge for 
students. This process has also helped OUSD to incorporate other student data to improve the 
college readiness systems in the district. Perhaps most importantly, the policy also conveys a 
powerful message from the district: that all students are capable of going to college. 


[The institutionalization of] the PSAT for 10th graders sends a message to students 
that the district believes everyone can go to college. 
- Oakland Accelerates Team Member 


By the third year of the project, several members of the Oakland Accelerates team at OUSD cited 
the collaboration and coalition building between departments as a result of the program team 
stabilizing as one of the great successes. Interviewees mentioned how the Linked Learning office is 
interacting more with Curriculum and Instruction, Leadership Curriculum and Instruction is 
working with the high school office and full services community schools. The CR Specialists, 
Content Specialists, and program team members have developed a true collaborative partnership by 
understanding and valuing the strengths each individual brings. 


Prepared by Hatchuel Tabernik and Associates Page | 28 


| just want to say that |!am so proud of the people | work with. They are such an 
amazing team, instead of being discouraged by the weight of the lofty goals that 
were set for the i3 grant; they just look at what the next step needs to be. 

- Oakland Accelerates Team Member 


Another successful activity cited by those interviewed was the 9*" grade college and career plans. 
With a goal of 100% completion of CCPs, the team has increased the number of 9" grade students 
with a plan every year since the beginning of the grant. 


Looking towards sustainability, CR and Content Specialists have put successful systems in place 
for designing and implementing professional learning in schools. Interviewees expressed that with 
the support of College Board-supplied experts and program collaboration, Specialists have created 
systems that can continue sustainably into the future. Examples include data-driven modules for 
teachers, conversations around college transcripts, and processes and calendars for preparing 
schools for the aforementioned annual PSAT and College and Career Community Plans. 
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Appendix: Fidelity Analysis & Matrix 


College Board’s EXCELerator model aimed to work with schools to build infrastructure to support 
college readiness in all students via a multiphase data-driven decision-making process. Oakland 
Accelerates (OA) in OUSD was the first application of this model at a district-level. Measurement of 
implementation had specific requirements in order to meet the levels of rigor required by the 
Department of Education. Details for tracking, collecting and measuring fidelity across the four key 
components as defined in the research study can be found in Tables A2-A5. These goals differ from 
many of the Oakland Accelerates activities outlined earlier because the research study is primarily 
interested in the “intervention” which is defined as the activities College Board delivered as part of 
their EXCELerator 2.0 model. This intervention then hypothetically drove or resulted in the 
Oakland Accelerates activities. (See Figure 2 above which presents the Logic Model which drove the 
fidelity analysis.) 


To assess fidelity to implementation, HTA analyzed data from years one and two including 
interviews with key stakeholders, meeting notes, program documents, and “fidelity dashboards” in 
order to determine whether key components of the program were executed in a manner meeting the 
threshold to be considered implemented with fidelity. This was determined using a four-component 
fidelity matrix, or rubric, identifying each key component of the intervention, a range of 
implementation possibilities, and a threshold by which to determine whether each key component 
was implemented with fidelity or not. For example, the threshold for Professional Development 
opportunities provided annually to teachers is five trainings or more. If College Board provided less 
than five trainings, that key component of the intervention did not meet fidelity. 


Table A1: Overview of Fidelity to Implementation 
Key Component of Oakland Accelerates 2012-13 2013-14 
District Diagnostic N/A 


AP Professional Development 
College Readiness Specialist Support & Coaching 


Capacity & Policy Guidance 


1 = Activities were implemented as intended 


In the first school year, only one of the four components (“District Diagnosis”) was implemented 
with fidelity. By the second school year, two of three components" (“Professional development of 
AP teachers”, and “Support & coaching of college readiness specialists”) reached the threshold for 
fidelity. “Capacity and policy guidance” was the only component that did not meet the threshold for 
fidelity in either year, in part because parent outreach goals were not reached and in part because 
turnover in district leadership permanently delayed adoption of recommended policies. College 
Board’s participation ended in the second year. See Table Al for an overview and Tables A2- A5 for 
breakdowns of each component. 


It is important to note that the fidelity matrix provides only a narrow picture of the OA 
implementation. After reviewing three years of data, it is apparent there were several barriers to a 


1! The District Diagnostic component is only applicable to Year One. 
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fully-realized implementation, in part because predicted mediators did not occur as expected. For 
example, teachers and principals were not as involved as intentionally as OA program leaders would 
have liked, often resulting in staff resistance to suggested changes to their instructional practice. In 
addition, turnover in program and district leadership as well as at College Board meant OA program 
leaders were required to reconsider and retread previous decisions which impacted the ability to 
implement key components as well as mediated components during project implementation period. 
These barriers to implementation were described in detail in the main narrative of the 
implementation report. 


Prepared by Hatchuel Tabernik and Associates Page | A-2 


Table A2. Fidelity Matrix Component 1: District Diagnostic 


Tate l(orzhoyg 


District Diagnostics 
conducted 


District Diagnostic 
results shared 


Key Component 
Fidelity Threshold 


Operational 
Definition/ 
Criteria 


College Board 
performs District 
Diagnostic at 
2012-13 


CB shares written 
results through 
written and 
presentation 
format 


Data Source 


District 
Diagnostic 


Agenda notes 
of 
presentation 
to leadership 


IDF) i) 
Core} (=Xeia (ola) 
re ETA) 


HTA to collect 
District 
Diagnostic 
from College 
Board 


HTA to collect 
notes from 
OUSD and 
College Board 


With Fidelity: sum of indicator fidelity scores = 2 


Range: 0-2 


Level 1 
Fidelity Score 
Criterion 

(Tate livife [¥E=))) 


Level 2 Fidelity Score 
(District) 


0 = CB does not 
perform a District 
Diagnostic 


1=CB performsa 


District Diagnostic 


0 =CB does not 
share results 


1=CB shares results 


Activity 
Level 
Fidelity 
Score 
Criterion 


With 

fidelity = 1 
on district 
level score 


With 

fidelity = 1 
on district 
level score 


2012-13 


Score 


2 


With 
Fidelity 


2013-14 
Score 


NA 


NA 
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Table A3. Fidelity Matrix Component 2: AP Teacher Professional Development 


Indicator 


College Board 
provides teachers 
professional 
development trainings 


Follow up 
visits/observations 
after AP Teacher 
training 


Key Component 
Fidelity Threshold 


Operational 
Definition/ 
Criteria 


CB provides 5 
professional 
development 
trainings: AP 
Teacher Training, 
AP Content 
Coaching, Vertical 
Teams, AP Annual 
and Equity 
Conference, and 
SAT Writing 
Workshop 


CB (or CB-trained 
OUSD coaches) 
conducts 3 in- 
class observations 
and provides 
additional 
coaching where 
necessary 


Data Source 


Progress 
Dashboard 


Progress 
Dashboard and 


IDF) i) 
(fe) | (=e d fo} 
ETA) 


CB to send 
HTA Progress 
Dashboard 
quarterly 


CB to send 
HTA Progress 
Dashboard 
quarterly 


Level 1 
Fidelity Score 
Criterion 
(Individual) 


0 = eligible 
teachers 
receive <3 
follow up 
visits 


Level 2 Fidelity Score 


0 =CB provides <5 
training sessions 


1=CB provides >5 
training sessions 


0= >75% of eligible 
teachers meet 
individual level 
criterion 


Observation 
records 


HTA to 
collect 
observation 
records from 
OUSD and CB 
quarterly 


With Fidelity = sum of indicator fidelity scores = 2 


Range: 0 - 2 


1 = eligible 
teachers 
receive > 3 
follow up 
visits 


1=>75% of eligible 
teachers meet 
individual level 
criterion 


Activity 
Level 
Fidelity 
Score 
Criterion 


2012-13 
Score 


With 

fidelity = 1 
on district 
level score 


With 

fidelity = 1 
on district 
level score 


0 


Without 
Fidelity 


2 


With 
Fidelity 
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Table A4. Fidelity Matrix Component 3: College Readiness Specialist Coaching & Support 


Indicator 


CB provides CRS 
Training 


College Board meets 
with CRS 


Operational 
Definition/ 
Criteria 


College Board 
provides 2 
trainings: Fall 
Counselor 
Workshop and SAT 
Tools Training ( 
and College 
Readiness 101-- 
Year one only) 


35 weekly group 
check-in meetings 


Data Source 


Progress 
Dashboard 


Progress 
Dashboard 


IDF) i) 
(fo) | (=r d (oa) 
plan 


CB to send 
HTA Progress 
Dashboard 
quarterly 


CB to send 
HTA Progress 
Dashboard 
quarterly 


Level 1 
Fidelity 
Rofo) f=) 
Criterion 
(Tate liVife [VE] 


Level 2 Fidelity Score 


O =CB provides <2 
training sessions 


1 =CB provides 2 
training sessions 


0 =< 25 of meetings 
occurred 


1 => 25 of meetings 
occurred with at least 
12 each semester 


Activity 
Level 
Fidelity 
Score 
Criterion 


With 
fidelity = 
lon 
district 
level 
score 


With 
fidelity = 
1on 
district 
level 
score 


2013-14 
AYofo) =) 


Technical support 
around College 
Board programs 


CB provides 
information about 
CB programs that 
CRS need/ request 
to work with their 
sites 


Progress 
Dashboard 


CB to send 
HTA Progress 
Dashboard 
quarterly 


0 = <100% of CRS were 
provided support when 
requested 


1 = 100% of CRS were 
provided support when 
requested 


With 
fidelity = 
1on 
district 
level 
score 
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Table A4. Fidelity Matrix Component 3: College Readiness Specialist Coaching & Support 


Indicator 


Resources and 
materials provided 
to CRS 


Key Component 
Fidelity Threshold 


Operational 
Definition/ 
Criteria 


Implementation 
Mer provides each 
CRS with a 
resource binder 
related to PSAT, 
AP, SAT. Refers 
them to resources 
as necessary. 


With Fidelity = sum of indicator fidelity scores >3 and must achieve “with fidelity” on CB provides CRS 
training 


Range: 0-4 


Data Source 


Progress 
Dashboard 
and Resource 
Log 


DF) i] 
(ore) |(=Yoiu (ola) 
plan 


CB to send 
HTA Progress 
Dashboard 
and Resource 
Log quarterly 


Level 1 
Fidelity 
Score 
Criterion 
(Individual) 


Level 2 Fidelity Score 


0 = <100% of CRS were 
given binder 


1 = 100% of CRS were 
given a binder and 
were referred to PSAT, 
AP, SAT resources/ 
materials as needed 


Activity 
Level 
Fidelity 
Score 
Criterion 


With 
fidelity = 
1on 
district 
level 
score 


2 


Without 
Fidelity 


3 


With 
Fidelity 
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Table AS. Fidelity Matrix Component 4: Capacity and Policy Guidance 


Titel Cor} 


CB provides District 
Staff Training 


College Board meets 
with District 
Leadership 


CB provides ongoing 
Data Coaching to 
OUSD Data Team 
(district officials and 


Operational 
Definition/ 
Criteria 


College Board 
offers 6 trainings: 
PSAT Coordinators 
Training, PSAT 
Tools Training, 
“Understanding 
the PSAT” Training, 
SOAS Training, AP 
Coordinators 
Training, AP 
Potential 


59 in-person 
meetings or 
phone/email 
capacity/policy 
guidance updates. 
(5 groups: LCI 
quarterly, District 
Leadership/ 
cabinet, once per 
semester, OUSD 
Project Director, 
weekly, OUSD high 
school network 
officer, monthly, 
Principals, 
monthly) 


10 days of data 
support 


Data Source 


Progress 
Dashboard 


Progress 
Dashboard 


Progress 
Dashboard 


Data 
Core} (-Xeia (ola) 
re ETA) 


CB to send 
HTA Progress 
Dashboard 
quarterly 


CB to send 
HTA Progress 
Dashboard 
quarterly 


CB to send 
HTA Progress 
Dashboard 
quarterly 


Level 1 
Fidelity 
Score 
Criterion 
(Individual) 


Level 2 Fidelity Score 


0 =CB provides < 6 
training sessions 


1 =CB provides 6 
training sessions 


0 =< 47 meetings 
occurred 


1 =>47 meetings 
occurred. At least 2 
with LCI staff, 2 with 
District 
Leadership/cabinet 
staff, 27 with Project 
Director, 8 with high 
school network officer 
and 8 with Principals. 


0 =<8 days 


of coaching occurred 


Activity 
Level 
Fidelity 
Score 
Criterion 


With 
fidelity = 
1 on 
district 
level 
fidelity 
score 


With 
fidelity = 
1on 
district 
level 
score 


With 
fidelity = 
1on 
district 


2012-13 
NYofo) =) 
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Table AS. Fidelity Matrix Component 4: Capacity and Policy Guidance 


Level 1 Activity 
Operational Data Fidelity Level 
Indicator Definition/ Data Source (efo} | (=Yoi doy} Rofo) f=) Level 2 Fidelity Score Fidelity 
Criteria re ETA) Criterion Score 
(Individual) Criterion 


principals) level 
score 
1 => 8 days of coaching 
occurred with at least 
one per quarter 


0 =CB provides < 2 
trainings to parent : 
CB conducts two Pe : el 
CB supports parent CB to send Fidelity = 
lee Parent Leader 
capacity in student ‘oiniee ea Progress HTA Progress lon 
college and career Dashboard Dashboard district 
readiness belt ee andicaneer uarterl level 
readiness q ¥ 1 =CB provides 2 aera 
trainings to parent 
leaders 


2012-13 
Score 


Progress 0 = CB distributes 

Dashboard resources at < 3 OUSD 

and CB to send parent events With 
HTA Progress Fidelity = 
Dashboard lon 


CB to have a 

CB provides college resource and 
readiness resources information table Lee 

and Event district 


directly to parents at four OUSD Calendars 1 = CB distributes 


quarterly 


level 
score 


parent events resources at >3 OUSD 


arent events 
Event P 


Calendars 


Key Component With Fidelity: sum of indicator fidelity scores 4 and must achieve “with fidelity” on CB provides district staff 
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Table AS. Fidelity Matrix Component 4: Capacity and Policy Guidance 


Level 1 Activity 
Operational Data Fidelity Level 
Indicator Definition/ Data Source (efo} | =Yoi doy) Rofo) f=) Level 2 Fidelity Score Fidelity 
Criteria plan Criterion Score 
(Individual) Criterion 


Fidelity Threshold training 


Without Without 


Range: 0-5 Fidelity Fidelity 
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